Your JSON Is Valid but Your Data Is Wrong: Five Failure Modes LLM Structured Outputs Won’t Catch

Constrained decoding solved a real problem. Before grammar-based methods like Outlines and SGLang, getting valid JSON from a language model was a retry loop. You prompted, parsed, caught the trailing comma, reprompted. Constrained decoding ended that: force token selection through a finite-state machine, and every output parses.

Teams adopted it fast. Schema compliance hit near 100%. And then a quiet assumption crept into production codebases: if the JSON validates against the schema, the data is correct.

BAML’s benchmarks say otherwise. On function-calling tasks, unconstrained generation with post-hoc parsing reached 93.63% accuracy; constrained decoding on the same model scored 91.37%. The always-valid JSON was less accurate than the sometimes-broken JSON.

I started tracking this after a classification pipeline I built began returning plausible but fabricated values on roughly one in twelve runs. The JSON always parsed. Pydantic never complained. It took weeks to notice, because every downstream check was structural.

Five failure modes keep showing up. They all produce schema-valid output that breaks your pipeline silently:

1. Enum hallucination: valid enum, wrong meaning

2. Confident fabrication: plausible values in free-text fields

3. Cross-field contradiction: fields valid individually, impossible together

4. Distributional collapse: convergence on safe defaults

5. Array hallucination: fabricated entries instead of empty arrays

Constrained Decoding: The Problem It Actually Solved

Structured output used to mean hoping the model behaved. Constrained decoding was built to fix that, and it did, just not the whole problem.

The progression was real. Prompt-and-pray JSON, where you appended “respond in JSON format” and crossed your fingers, gave way to regex-guided generation (LMQL), then to grammar-based constrained decoding.

XGrammar, now the default backend for vLLM and TensorRT-LLM, adds near-zero overhead per token. The syntax problem is solved.

But solving syntax created a blind spot. Schema validation checks whether a field is typed correctly: a string is a string, a number is a number. It says nothing about whether that string or number is correct.

A lock on a filing cabinet keeps the drawers organized. It says nothing about whether the papers inside are accurate. Schema validation works the same way.

The reason this matters: forcing a model into a strict output format costs it something. It has to spend part of its attention on staying inside the format, instead of spending all of it on getting the actual answer right.

Lee et al. measured that cost directly. Across open-weight models, forcing structured output formats produced a 3-to-9 percentage point accuracy drop. On math reasoning tasks specifically, where getting the reasoning right matters more than the format, the gap exceeded 15 percentage points.

Tam et al. found the same pattern from a different angle: the stricter the formatting rules, the worse the reasoning got. The format is not free, and most teams aren’t accounting for that cost.

Five Failure Modes: What Survives Your Schema

Schema validation catches type errors. It does not catch these five failure modes, because each one produces output that is structurally valid and substantively wrong.

Enum hallucination. The model picks a valid enum value that is semantically wrong for the input. Consider a priority enum of ["low", "normal", "high", "urgent"]: the grammar guarantees one of those four values, but it does not weight them by input context. The model can return “urgent” on a routine request or “low” on a critical one, and the schema will accept both.

Confident fabrication. Free-text fields return plausible but invented data. BAML demonstrated this by submitting a photo of an elephant as a receipt: constrained decoding returned a complete, schema-valid expense report instead of refusing. Constrained decoding eliminates the model’s ability to refuse or express uncertainty. The schema requires a value; the model provides one, whether or not the input supports it.

Cross-field contradiction. Schema validation checks each field in isolation. It never checks whether the fields agree with each other. A sentiment extractor can return {"sentiment": "positive", "score": 0.1}, a positive label with a score close to zero, which should mean negative. A date parser can return {"start": "2026-03-15", "end": "2026-03-10"}, an end date before the start date. 

Both outputs pass every individual field’s validation. Neither makes sense once you look at the record as a whole, and no single-field validator is built to catch that, because the constraint lives between fields, not inside any one of them.

Distributional collapse. The model converges on safe, generic values across different inputs. Constrained decoding biases toward high-probability tokens within the valid set, and “safe” defaults (0.95, “medium”, “general”) carry higher base probability than context-specific values.

I caught this when confidence scores in a classification pipeline flatlined at 0.98 for three weeks. Collin Wilkins documents a similar case where confidence was 0.99 on every output, including gibberish. Every record had valid types, correct enums, reasonable numbers. The distribution had stopped moving, and nothing alarmed because each individual output was structurally correct.

Array hallucination. Models resist returning empty arrays. Under constrained decoding, [] is a low-probability token sequence because the grammar weights object-producing paths more heavily than the empty-array path. When a schema requires an items field of type array, the model fabricates entries rather than returning nothing.

In extraction tasks, this produces phantom results: your pipeline reports “found 3 matches” when the correct answer is zero.

Five failure modes mapped to their observable signals, root causes, and detection strategies.

The Validation Trap: Why More Rules Won’t Fix This

The first instinct is to write more validation rules. For known patterns, that works. A Pydantic model_validator catches start_date > end_date. A custom check flags sentiment-score mismatches. You can build cross-field constraints for every failure you’ve already seen.

The problem is the failures you haven’t seen. Structural correctness is closed: you can enumerate every valid JSON shape for a given schema. Semantic correctness is open-ended. You can’t write a rule for a wrong answer you haven’t encountered yet. In production, the model finds new ways to be wrong faster than you write validators, and each new validator only covers the last bug.

There’s a contrarian argument for resampling over constraining that deserves weight here. Instead of forcing the model’s output into a grammar as it generates, let it write freely, then check the result: a parser validates the output against the schema, and if it fails, the model simply generates again. This is resampling. Free-form generation lets the model reason without format pressure; the format check happens after, not during. BAML’s benchmarks show parse-and-retry outperforming constrained decoding by over 2 percentage points on the same model.

You trade guaranteed first-pass parse success for higher accuracy when the output does parse. Whether that trade-off holds at high volume, where retries compound latency, is still an open question.

But the deeper problem cuts across both approaches. Structured output hides uncertainty. When the schema requires a value, the model fills it in. A risk_score field always gets a number, even when the model has no basis for the assessment. A summary field always gets text, even when the input contains nothing to summarize.

There is no standard mechanism for the model to express “I do not know” or “this field does not apply to this input.” The schema is a forcing function, and wrong answers emerge with the same confidence as right ones.

After the third time I caught schema-valid-but-wrong output in production, I stopped treating schema validation as a quality gate and started layering semantic checks on top. Schema compliance is the floor, not the ceiling.

Three-Layer Defense: Schema, Semantics, Uncertainty

Layer 1: Schema and structural validation. This is what you already have: Pydantic, JSON Schema, Zod. It catches type errors, missing fields, and syntactically invalid enum values. Keep it. It solves the syntax problem well.

Layer 2: Semantic validators. Cross-field constraint functions that encode business logic: “if sentiment is positive, score must exceed 0.5.” Distribution monitors that track field-value entropy over time. When entropy drops below a threshold, you catch distributional collapse before downstream metrics drift.

Periodic sample audits on outputs from ambiguous or edge-case inputs catch confident fabrication. This layer requires domain knowledge and ongoing maintenance, but it covers most of the five failure modes, because it checks meaning, not shape.

Layer 3: Uncertainty surfacing. Add an optional confidence field alongside every extracted value, so the model can express what it doesn’t know. Cleanlab’s CONSTRUCT benchmark shows that per-field trustworthiness scoring detects errors in structured outputs from GPT-5 and Gemini with higher precision than prompt-level confidence estimates.

For high-stakes fields, add LLM-as-judge verification: a second model call that evaluates whether the extracted value is supported by the input. The cost is latency. The payoff is catching failures before they reach your pipeline.

Most teams have Layer 1 only. Adding Layer 2 catches the majority of silent failures. Layer 3 is for fields where a wrong answer costs more than the extra latency.

Three Signals: When Your Pipeline Is Silently Failing

You will not catch these failures by inspecting individual outputs. The signals are statistical.

• Output entropy is dropping. If a field that should vary across inputs starts clustering around one or two values, the model is collapsing to safe defaults. Plot value distributions weekly.

• No output is ever empty. If an array field that should sometimes be empty never returns [], the model is fabricating entries. Check the empty-array rate against your expected base rate.

• Downstream metrics drift without upstream changes. If your business metrics shift but the model version, prompt, and schema haven’t changed, the model’s semantic accuracy may have degraded while structural compliance stayed perfect. This is the hardest signal to attribute, and it’s often the first one.

Conclusion: Schema Is the Floor, Not the Ceiling

Default to constrained decoding for parse reliability. Build the semantic checks it was never designed to provide.

Schema validation will never tell you the data is right. It tells you the data is shaped correctly. The gap between those two claims is where these five failure modes live.

Further Reading

• Structured Outputs Create False Confidence (BAML’s benchmark on accuracy degradation under constrained decoding)

• Let Me Speak Freely? (EMNLP 2024 study on reasoning decline under format restrictions)

• The Format Tax (measuring the 3-9pp accuracy cost of structured output formats)

• JSONSchemaBench (10K real-world schemas across six constrained-decoding frameworks)

• Cleanlab CONSTRUCT (per-field trustworthiness scoring for structured LLM outputs)

• From Hallucination to Structure Snowballing (how constrained decoding triggers formatting traps during self-correction)

···

Thanks for reading. I’m Mostafa Ibrahim, founder of Codecontent, a developer-first technical content agency. I write about agentic systems, RAG, and production AI. If you’d like to stay in touch or discuss the ideas in this article, you can find me on LinkedIn here.