Skip to content
Raunak Sood
 
← Back
Writing · September 2026

The schema is not the cost

If you run an LLM behind a JSON schema, you have probably worried about what the grammar is doing to the model. I went looking for that cost in a specific place and found the opposite.

Items the model was asked to find
playing · click a value to take over
with a JSON schema
0.440
0 of 1 recovered
0.5 items emitted · precision 0.917
free generation
0.640
1 of 1 recovered
0.8 items emitted · precision 0.882

At k = 1 free generation is ahead by 20.0 points. The schema is costing recall here, which is what everyone expects it to do.

Constrained decoding masks tokens the model wants, and the reported costs are real: several percentage points of accuracy across open-weight models, and at least one function-calling result where unconstrained generation with post-hoc parsing beat constrained decoding outright.

The question

Ask a model to pull every named counterparty out of a 10-K. Does it get worse at that as the number of counterparties grows, holding the document length and the schema fixed? Call that number k.

It is not a new question in spirit. Multi-needle NIAH already reports that retrieval falls as needle count rises. Nobody had measured the same axis for structured extraction under a grammar, with input length held constant, reported as a slope rather than as “it declines.”

I had a specific reason to care. In my own extraction work I had written down constrained decoding as the strongest competing explanation for any cardinality effect I might find. I recorded it as unfalsified and moved on.

Building a ruler that works

You cannot answer this by annotating documents, because k is the treatment. If it comes from whichever filings you happened to sample and a human has to recover it, it is measured with error that correlates with the outcome, because an annotator's attention flags on the long enumerations where the effect should live.

So I built the documents instead. Take verbatim-validated sentences from real filings, each carrying exactly one named company. Inject k of them at random positions into filler drawn from the same corpus with every capitalised name stripped out. Pad to a constant length. Gold is the set you injected, so k is exact and free.

Getting that design to work took four builds, and the failures taught me more than the design did.

The first pilot came back 0.333. One needle, from a sentence the extractor itself produced in the wild, found a third of the time. Only then did I add the gate that should have existed from the start: at k=1, recall must clear 0.80 or no slope is interpretable. The pre-registration records it as added after that failure, not before. After a fix it came back 0.400, then 0.400 again.

The first cause was that my injected sentences named their own original filer in the third person, while the document was attributed to a placeholder company. The model was being asked who ACME deals with and correctly declining to say Lockheed. The second round found four more defects at once: an all-caps ticker the name-detector missed, paragraph chunking that sliced sentences in half at fixed offsets, gold items that were not company names, and a matching rule strict enough to score Komatsu against Komatsu Cummins Chile, Ltda. as a miss.

I fixed all four and recall was still 0.400. At that point I stopped patching things I could see and ran a controlled comparison instead.

1,500
0.667
4,000
0.583
14,000
0.417
The same sentence, the same single item, only the amount of surrounding filler changing. Dilution alone is worth 0.25 recall, which is why the k = 1 sanity gate failed three times before the cause was measured rather than guessed.

Dilution alone was worth 0.25 recall. I had built the filler by selecting prose for containing no company names, which made it far more inert than any real prompt. The gate kept failing, whatever I did to the bank or the chunking, because I had buried one sentence in a haystack far emptier than anything the model sees in practice.

It also exposed a constraint. Count, density and length are mechanically linked: density = (count × span length) / total length. Fix any two and the third follows, so every study of item count picks two and inherits a confound in the third. Fix length and let density rise with k, as I did, and high-k documents get a density advantage that partially cancels the effect, so your measurement is a lower bound. Fix density and let length grow, as multi-needle NIAH does, and length suppresses high k, giving an upper bound. There is no third option.

The result

qwen3:14b, 32k context, 75 matched documents per arm (25 at each of k = 1, 4, 16), no response within a tenth of the 24,000-token ceiling (the longest is 903 tokens). One caveat first: the k=1 gate above was never met. The final run came in at 0.440 constrained and 0.640 unconstrained against the 0.80 I had set, and the pre-registration said to abandon the constructed design at that point. I ran the comparison anyway. So the absolute levels describe the instrument, not the model, and only the between-arm contrast on identical documents is readable.

0.40.50.6k = 1k = 4k = 16schemafree
Recall against the number of items to emit, same 75 documents per arm. The arms cross between k = 1 and k = 4. At k = 16 the gap is 0.045, p = 0.026. Hover a level for the pair.

First, a correction to what this post said until October 2026. It reported a 0.214 drop under free generation and called it a rejection. Six of the 75 free-form answers had been scored as empty because my tolerant parser rejected JSON whose strings contained quotation marks the model had copied from the filings, like Express Scripts, Inc. ("Express Scripts"). Those six complete answers held 39 items, 38 correct. The scorer now repairs interior quotes, fails closed if that does not yield valid JSON, and has unit tests. Re-scored, the free-form drop is 0.131 at p = 0.027: the rank test clears, the size does not, and the pre-registered rule rejects for neither arm.

Paired on the same documents, unconstrained leads by 0.200 at k=1 and trails by 0.045 at k=16. The interaction test, which I did not pre-specify, puts the gap at −0.061 per doubling of k (p = 0.020), carried by k=1, where the constrained arm returns an empty list on 13 of 25 documents. Remove those and the slope is gone (p = 0.17): above one item, both arms lose recall at about the same rate.

So the grammar is not the cost, and on this evidence it is not the cure either. What it buys is validity. Without a grammar, four of 25 answers at k=16 are not valid JSON as written, and a scorer without a repair step reads that as zero recall, which is how this post came to report a result it did not have.

The emission counts: at k=16 the constrained arm emits 9.4 items of the 16 available; the unconstrained arm emits 8.5, at the same precision (0.946 and 0.953). Both stop well short. An earlier version of this post said a half-filled array is not a valid parse, so the schema forces the model on. That is false: the schema is a bare list with no minimum length, and most of the emission gap I reported was the parser defect. The experiment that would test a grammar-level stopping story, a minimum item count in the schema, has not been run.

That also kills a number I believed earlier in the week. On a 30B mixture-of-experts model, unconstrained generation cost 18× the tokens. I reported that as a fact about removing the grammar, and it was wrong: that model was writing 7,000 to 12,000 tokens against an 8,192-token context window, so the arm was measuring a wall. On a model whose generation fits, unconstrained costs 1.1× at k=1 and 0.7× at k=16. It is cheaper and worse.

What I would take from this

For anyone shipping structured extraction: the schema is not the thing costing you recall at high item counts, and what it buys you is output you can parse. Free-form generation with post-hoc parsing costs no fewer tokens here and breaks the format more often as the list grows. Whether it costs recall is not settled at 25 documents per level. If you are considering it to escape grammar masking, measure the item-count axis and test your parser before you switch.

For anyone evaluating extraction: a single F1 over a benchmark whose k distribution is unstated is not comparable to another benchmark's F1. And if you measure the axis, say which two of count, density and length you fixed, because that choice decides whether your number is an upper or a lower bound.

The methodological lesson is narrower. I had recorded constrained decoding as the strongest competing explanation and left it unfalsified. Recording it ruled nothing out, and everything in this post came from testing it.

One more correction to the record. The pre-registration originally specified a different second factor, model size against think/no-think. The think factor proved inert on September 11 and was replaced by constrained-vs-unconstrained decoding before the unconstrained arm was run; the rejection rule did not change. The repository's first commit contains both the pre-registration and the results, so the ordering rests on the document's own dated amendments, not on commit history.

Code, pre-registration, evaluation set and all 450 raw model responses: cardinality-eval. The result re-scores without a GPU.