Governed SQL analyst agent

The part that is actually hard

Getting a model to write SQL is not the interesting problem. In a warehouse where a wrong number moves money, the interesting problem is everything that decides whether the number it produced is allowed to reach a human, and whether anyone can reconstruct that decision afterwards.

So this agent is built around a boundary: the model scores, suggests and drafts, governed code decides. It cannot widen its own access, it cannot skip its own checks, and every turn it takes is replayable from the log.

One question, end to end

The ordering is the point. Validation is static and cheap, so it runs before execution and an unsafe query never reaches the database. The cross-check runs after the answer is written, because the number being checked is the one the agent is about to state.

question
  -> inbound screening      instruction-shaped input is refused and logged
  -> plan                   the model writes SQL and declares which canonical
                            metric its headline number is supposed to equal
  -> validate               AST: one statement, read-only, allowlisted relations,
                            no restricted columns, no file or network functions,
                            hard row cap injected
       rejected -> the rule is fed back to the model, up to 2 repairs
  -> execute                read-only connection, 15 second timeout
  -> narrate                the model writes the answer strictly from the rows
  -> checks                 empty result, truncation, source freshness,
                            negative counts, canonical cross-check
       any failure -> the number is not sent, the turn escalates
  -> answer + cited evidence + audit record

The query governance layer

Rules are enforced against the parsed syntax tree, not against the query text, which is why comment tricks and string mangling do not get through. The whole policy sits in one readable file, so a change to it is a single clear diff.

Defence is layered rather than stacked on the input screen. A prompt injection that slips past the screen still cannot produce a query that leaves the allowlist, and a model that decides to be helpful about email addresses still cannot select the column.

Checks before a number is sent

A failed check does not produce a hedged answer. It produces no number at all, plus a trace id for a human to open. And the model does not get the last word on whether the cross-check runs: when the query returns a single number whose alias names a canonical metric, the check is derived from the query itself. A control that only fires when the model remembers to ask for it is not a control.

What a reviewer sees

Every turn appends a full record to a decision audit log, and the supervisor console is generated from that log: rejected queries shown next to the rule that rejected them, and grading buttons whose output exports as evaluation data.

The dashboard the team steers by

The agent writes down every decision it makes, so its own operating numbers are a query away. This dashboard is generated from that log plus the evaluation history: decision volume, why turns did not produce a number, how each check behaved, cross-check drift against the 0.5% threshold, latency, suite history, and the queue of turns a human should still look at.

Both surfaces are live, open them and click around:

performance dashboardsupervisor console

The console is the review surface: expand any turn to see the SQL that was proposed, the rule that rejected it, the rows, the checks, and grade it. The grades export as JSONL, which is the format the evaluation cases are written in.

Evaluation

For a non-deterministic system a single green run is not evidence, so the suite reports per-case stability across repeated runs. Every run is appended to a history file, so the number below can be re-derived rather than taken on trust.

All five misses are the same case: an average per active player per day, which the model still gets wrong about a third of the time. It either cannot write the query within its three attempts or produces a number a check rejects. It has never sent a wrong number for it: it sends no number. The case stays in the suite at its real pass rate rather than being quietly reworded until it goes green. The other fifteen have not missed once in sixteen runs.

What the suite caught during development, each of which became a fix rather than a note:

• The agent filtered a timestamp column against a bare end date, silently dropping the last day of every range. Caught by the cross-check at 10.9% drift.
• It summed all withdrawals where the canonical view counts settled ones only. That is how a metric definition drifts in production. Caught at 29.6% drift and kept as a permanent regression case.
• It refused a perfectly answerable question because an instruction to prefer the reporting views read as a prohibition on base tables. Over-refusal is a failure and the suite grades it as one.
• Once the suite started grading whether the cross-check actually ran, rather than only whether the answer looked right, one case dropped to 1 out of 5: the agent answered correctly but skipped the declaration, so the number went out uncorroborated. The fix was not a sterner prompt. The derivation moved into deterministic code and the case went back to 5 out of 5.
• Then the cross-check itself became the bug. Running a bank of 40 real questions through the agent surfaced four answers blocked for nonsense reasons: a bonus figure compared against game payouts, a count of rejected withdrawals compared against a euro total, the largest single withdrawal reported as 1862% away from the monthly figure. The check was matching on column names. It now also requires that the metric comes from a relation it is defined on, that the aggregate is the same kind (a maximum is not a total, an average per player is not a count), and that the query is not answering a filtered slice. All four are permanent regression cases.
• Net deposits came back inverted, withdrawals minus deposits, across a whole batch of runs. The fix was not a prompt patch: the metric definitions moved into the warehouse catalog the agent reads, next to the columns they describe, because that is where a definition belongs.
• A query that passed validation and then failed inside the database used to end the turn. A database error is a repairable mistake like any other, so it now goes back to the model the same way a rejected query does.
• Two of the graded cases still missed at an earlier stage. Once the agent's net deposit figure disagreed with the canonical view and the cross-check blocked it, so the user got no number rather than a wrong one. Once a correct answer failed a grading assertion and did not reproduce in twelve further runs. Both are on the record, because a suite that only ever reports green is a suite nobody is learning from.

What is real and what is simplified

All data in this project is synthetic and generated from a fixed seed. No real company, customer or transaction data is used anywhere in it. Built in August 2026. Source code is available on request.