Skip to content
Raunak Sood
 
← Back
LLM evaluation · Python

The schema is not the cost

Ask a model to pull sixteen things out of a document instead of one, and it finds a somewhat smaller share of them, with or without a form. What the form changes, on this model, is whether the answer is valid at all: four of twenty-five free-form answers at sixteen items are not valid JSON as written.

What I proposed
A way to measure how extraction recall changes with the number of items, with no annotators: build the documents yourself from verbatim filing sentences, so the number of items is exact and the answer key is free. And a constraint every such study inherits: item count, item density and document length cannot all be held fixed, so each design picks two and should say which.
What it produced
A pilot on one model, 25 documents per level. Neither arm meets the test I fixed in advance: free-form recall falls 0.131 from 1 to 16 items (the rule needs 0.15) and schema-constrained recall does not fall. The gap between arms is carried by the one-item documents and vanishes without them. What the schema demonstrably buys is validity: 25 of 25, 23 of 25, 21 of 25 free-form answers are valid JSON as the list grows; 75 of 75 under the schema. An earlier version of this page reported a rejection; that was a scorer bug, fixed and disclosed.

The problem

When you ask a language model for structured output (a list of records instead of a paragraph), you usually hand it a schema and force the answer to fit. The schema guarantees you get valid JSON back.

It also blocks the model from saying what it wanted to say. That has a known cost: several percentage points of accuracy across open models, and at least one published case where letting the model write freely and parsing afterwards beat the schema outright. So the folklore is that grammars make models dumber.

I had a specific reason to check. In my extraction work I'd written down constrained decoding as the most likely alternative explanation for something I was seeing, and then never tested it.

What I did

The hard part is knowing the right answer. If you take real documents and have someone mark up every company mentioned, you get labels that are expensive and unreliable, and an annotator's attention drifts on the long lists where the effect should show up.

So I built the documents instead. Take sentences from real SEC filings, each one naming exactly one company, each verified to appear word-for-word in the source. Drop k of them at random positions into filler text drawn from the same filings with every company name stripped out. Pad everything to the same length.

Now the right answer is whatever I put in. No annotation, and k is exact.

Items the model was asked to find
playing · click a value to take over
with a JSON schema
0.440
0 of 1 recovered
0.5 items emitted · precision 0.917
free generation
0.640
1 of 1 recovered
0.8 items emitted · precision 0.882

At k = 1 free generation is ahead by 20.0 points. The schema is costing recall here, which is what everyone expects it to do.

Every number is measured. The three stops are the three levels I ran, 25 documents each and 75 per arm, with nothing interpolated between them.

What I found

Under the rule I fixed in advance, neither arm clears the bar. Writing freely, recall falls 0.131 from one item to sixteen on the same documents, short of the 0.15 the rule requires, and most of that fall is which items happened to be drawn at one item, not the count (with item fixed effects the slope is near zero). Under a schema, recall at sixteen items is higher than at one (0.554 vs 0.440), though it dips from 0.640 at four.

0.40.50.6k = 1k = 4k = 16schemafree
Recall against the number of items to emit, same 75 documents per arm. The arms cross between k = 1 and k = 4. At k = 16 the gap is 0.045, p = 0.026. Hover a level for the pair.

The test was fixed in advance: to count, recall had to drop at least 0.15 from the short lists to the long ones, and clear p < 0.05. Free generation drops 0.131 at p = 0.027, which clears the second condition and not the first. The schema drops nothing; recall rises, p = 0.73. An earlier version of this page said free generation dropped 0.214 and rejected the null. Six of its 75 answers had been scored as empty because my parser choked on quotation marks the model copied from the filings, such as Express Scripts, Inc. ("Express Scripts"). Those six answers held 39 items, 38 of them right. The scorer now repairs that, and every number here is from the repaired scoring.

What the schema does change is whether the answer is valid. Without it, 25 of 25 answers at one item are valid JSON as written, 23 of 25 at four, 21 of 25 at sixteen. Under the schema, 75 of 75. The failed answers are complete. They break the format by quoting a company's own short name inside a string. That is the one effect of the grammar this study can show with an artifact. At sixteen available, the schema emits 9.4 items and free generation 8.5, at the same precision, so both stop well short. The schema is a plain list with no minimum length, the constrained arm returns nothing at all on 13 of the 25 one-item documents, and the experiment that would test a grammar-level stopping story, a minimum-length rule, has not been run.

The test the paper did not run at first is whether the gap between arms grows with the list. It does, by 0.061 per doubling (p = 0.020). But drop the one-item documents where the schema returned nothing and the slope is gone (p = 0.17). Above one item, both arms lose recall at about the same rate. The recall difference between arms rests on the shortest lists and is not a finding.

The part that took longest

The first pilot came back at 33%: one item hidden in one document, found a third of the time. Only then did I write down the check that should have come first, that the extractor must find a single item at least 80% of the time before any slope means anything. The pre-registration records the gate as added after that failure, not before it. The next two builds came back 40%, then 40% again after fixing four separate bugs I could see by reading the output.

Guessing wasn't working, so I measured instead: same sentence, same single item, with only the amount of surrounding filler changing.

1,500
0.667
4,000
0.583
14,000
0.417
The same sentence, the same single item, only the amount of surrounding filler changing. Dilution alone is worth 0.25 recall, which is why the k = 1 sanity gate failed three times before the cause was measured rather than guessed.

I had built the filler by picking text for having no company names in it, which made it far emptier than any real document. The sanity check kept failing through every bug fix because I had buried one sentence in a haystack nothing like the documents the model normally sees.

The check was never passed. The final run came in at 44% under the schema and 64% without it, against the 80% I had set, and my own pre-registration said to abandon the constructed design at that point. I ran the comparison anyway, and the paper says so. The consequence is that the absolute numbers describe the instrument, not the model, and only the comparison between arms on identical documents is readable.

That also surfaced a constraint. The number of items, how densely they're packed and how long the document is are not independent: fix any two and the third follows. So every study of this kind picks two and inherits a bias in the third, and which two you pick decides whether your answer is an over- or under-estimate.

Why it matters

If you run structured extraction in production, the schema is not what's costing you recall on long lists, and what it buys you is answers you can parse. Free-form generation with post-hoc parsing costs no fewer tokens here and produces output that has to be repaired before it can be scored; whether it costs recall is not settled at 25 documents per level. The other lesson is about the instrument: a tolerant parser is part of the measurement, and mine manufactured a result until it was tested like one.

The thing I'd keep: I had the alternative explanation in my own notes, marked as untested, and listing it there ruled nothing out. Everything on this page came from testing it.