where one agent generated a SQL query from a user’s natural-language question and a second agent — “the judge” — decided whether that query was safe and correct enough to execute automatically or whether it needed to stop and wait for a human to confirm before running.
For a few weeks it worked well enough that we stopped watching it closely. Then a query that should have been flagged got approved and ran. Nothing catastrophic happened, no data was lost, no write went to the wrong table, but the returned result was completely wrong. This happened because the query had silently dropped a filter clause the user’s question clearly implied and it took us a confused half-hour with an analyst to work out that the number they’d been given was simply incorrect. Just wrong and confidently presented as right by a judge that had approved it without hesitation.
That’s the incident that made us stop treating “LLM judge approved it” as equivalent to “it’s correct” and start treating the judge itself as a component that needed its own testing.
What’s in this article
- The incident and the first assumption
- What the judge was actually doing
- How a judge lies to you
- The fix that helped
- Calibrating the judge against humans
- Conclusion
The incident and the first assumption
My first assumption was that the judge had made a one-off mistake. LLMs aren’t perfectly reliable, a bad call happens, but you move on. So I re-ran the same query and the same judge prompt against it in isolation expecting either a repeat of the mistake or a correction.
It approved the query again with the same missing filter and same confidence. Whatever was happening wasn’t a random miss, it was a pattern the judge would reproduce reliably which meant something structural was going in the wrong direction.
To get to the roots of the issue I pulled a batch of past judge decisions and started comparing them against what a human reviewer would have said about the same queries and that’s when the pattern got specific enough to name.
What the judge was actually doing
The generator agent and the judge agent were built on the same underlying model. That wasn’t a design choice but it was just the model we’d standardised on for cost reasons across the whole pipeline. When I swapped in queries generated by a different model with the same task and schema, comparable quality on manual inspection, the judge got noticeably stricter. It caught issues in the other model’s queries that it had been letting through in its own model’s queries.
That’s self-preference bias, and it’s a documented behaviour in LLM-as-judge research, not something unique to our setup. What I hadn’t appreciated until it caused an actual incident is how consistent the effect can be. This wasn’t a judge that was randomly generous. It was a judge that was specifically generous toward outputs written in a style and structure it recognised as its own.
Here’s a stripped-down version of what our judge prompt looked like at the time, simplified for illustration, not the real production prompt:
JUDGE_PROMPT = """
You are reviewing a SQL query generated for the following user question.
Approve it for automatic execution, or flag it for human review.
User question: {question}
Generated SQL: {sql}
Schema: {schema}
Return JSON only:
{{"decision": "approve" | "flag_for_review", "reasoning": str}}
"""
def judge_query(question, sql, schema, client, model="gpt-4o"):
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": JUDGE_PROMPT.format(
question=question, sql=sql, schema=schema
)}],
temperature=0,
response_format={"type": "json_object"},
)
return json.loads(response.choices[0].message.content)Nothing about this prompt asks the judge to be lenient toward anything. The bias doesn’t come from the instructions, it comes from what the judge model recognises as “how a correct query is written” and if the generator writes queries in that same recognisable style the judge’s prior toward approving it shifts, independent of whether the query is actually right.
How a judge lies to you
Once I found self-preference bias in our judge, I started looking for other systematic ways the judge could be wrong and here are my findings.
Self-preference bias – An LLM acting as a judge can rate its own outputs, or outputs from its own model family, higher than others as happened in our case. The core driver of this bias is perplexity familiarity. LLMs tend to assign higher scores to text that is more predictable and has lower perplexity, which means it is generated by its own model family. Stronger models may exhibit this pattern even more because they are better at recognising their own stylistic fingerprints.
Verbosity bias – When you give a judge two answers to compare in which one is correct and concise and the other is same with extra context, the longer one wins more often than the shorter one. This was another bias pattern which we observed where the judge is giving higher scores to the SQL which contains more detailed comments rather than one which contains just the “exact” query. This bias repeats especially if your rubric has anything resembling “thoroughness” as a scoring dimension.
Position bias – For pairwise comparisons like “which of these two outputs is better”, the order you present the outputs shifts the outcome of your judge. Swap the order by keeping everything else identical and the judgment might flip. This is very easy to catch; run both orders and see if the verdict holds.
None of these three invalidate the approach. They simply rule out treating a judge’s score as an objective measurement, the way you’d treat a unit test passing or failing. It’s an opinion from a consistent but biased reviewer and you need to know its blind spots the way you’d learn a colleague’s blind spots before weighting their review the same as everyone else’s.
The fix that helped
The first fix was routing judgment to a model from a different family than the generator, a neutral third party rather than a reviewer sharing lineage with the thing it’s reviewing. This directly addressed the self-preference problem and it’s the change I’d like you to make first if you’re in a similar spot because it removes the mechanism instead of trying to statistically average it away.
# Simplified for illustration
def judge_query_neutral(question, sql, schema, generator_model, client):
# Never let the judge share a model family with the generator.
judge_model = "gemini-2-5-pro" if "gpt" in generator_model else "gpt-4o"
return judge_query(question, sql, schema, client, model=judge_model)It did nothing for verbosity bias because that bias isn’t about which lab trained the judge, it’s about what the rubric implicitly rewards. A neutral judge with a rubric that favours thorough-sounding output will still favour thorough-sounding output. I’d assumed going in that “pick a judge with no stake in either answer” would clean up bias generally. It fixed one specific bias and left the others exactly where they were, a genuinely useful thing to be wrong about, because it forced me to treat each bias as its own problem with its own fix instead of one problem with one patch.
What helped with verbosity was rewriting the rubric to explicitly instruct the judge to penalise unnecessary length and reward the same correct result in fewer words. Writing an example directly in the prompt showing a short correct query beating a longer over-hedged one helped here. Telling the judge what “good” looks like with a concrete example did more than any amount of generic “be objective” instruction do. This is a thing which I already knew about prompting models and had somehow not applied to my own judge prompt.
Calibrating the judge against humans
None of the above matters unless you check it against real human judgment on a held-out sample and keep checking, not once and done.
# Simplified for illustration
def measure_agreement(judge_decisions, human_decisions):
agreed = sum(
1 for j, h in zip(judge_decisions, human_decisions) if j == h
)
return agreed / len(judge_decisions)We pulled a sample of past queries the judge had already scored, then we asked a reviewer with schema knowledge to score the same sample without seeing the judge’s verdict first, and then we measured agreement. It came out in the low-to-mid 80s as a percentage which I’m flagging explicitly as a rough figure from our own pipeline at the time, not as a benchmark to expect on a different task or rubric.
The number itself isn’t the point. What it’s for is deciding what gets routed to a human automatically regardless of what the judge says. Queries that were plausible but subtly wrong, the exact category that caused the original incident, were consistently where judge-human agreement was the weakest. Those now get flagged for review by default independent of the judge’s own confidence because the judge’s confidence in that category had already proven itself unreliable.
This is also the actual case for using an LLM judge in a pipeline like this and it doesn’t get said often enough next to the bias caveats, it’s genuinely fast for a first-pass decision and it’s genuinely useful for bootstrapping a labeled dataset when you don’t have one yet; score a rough batch, have a human check the disagreements, and build your real evaluation set out of that loop, instead of writing every label by hand from nothing. The bias problems are real and they’re a reason to stop treating the judge’s output as a fact and start treating it as a first opinion that needs a second one on a schedule but not a reason to rip the approach out.
Conclusion
The query that started this didn’t slip through because our judge was badly prompted; it slipped through because we’d built a review step and quietly assumed it was neutral, when it was structurally inclined to approve of its own reflection. That’s the fix that actually mattered and it’s the one I’d tell anyone running an LLM judge in production to make first before touching a single word of the rubric. That one change closes the specific failure mode that caused the incident and it costs nothing beyond picking a different model for one API call.
The rubric rewrite and the human calibration loop are what turned the judge from “usually fine” into something we’re willing to gate automatic execution on. That means three things are now true that weren’t true before the incident: the judge and the generator are never from the same family, the rubric penalises verbosity explicitly instead of rewarding it by accident, and any query category where judge-human agreement has historically run weak is routed to a human by default regardless of what the judge says about it. None of that makes the judge infallible. It makes its failure modes known, bounded, and specifically the ones we’ve decided we can live with.
If there’s one thing worth taking from this is that an LLM judge earns the right to gate production actions the same way a new team member earns the right to approve their own pull requests: not on day one, and not just because it sounds confident when it explains its reasoning, but after you’ve watched it disagree with a human often enough in specific enough categories to know exactly where it can be trusted and where it can’t.