LLM Judge False Positives
in Table OCR Benchmarks
On OmniDocBench v1.6, a public document-parsing benchmark, recent table parsers clear 93 TEDS, the standard table-structure score. On a harder set of real-world tables in the same 2026 study, the strongest parser reached 85.03 (arXiv:2608.09842). The usual reading is that one benchmark is easier than the other. There is a second reading: between a parsed table and the leaderboard number sits a scoring instrument, and for many table benchmarks that instrument is an LLM judge, a model prompted to grade the parser's output, whose own error rate is never printed next to the score.
Key Takeaways
- An LLM judge, a model that grades the parser's output, matches human table scores at r = 0.93 where TEDS manages 0.68.
- At an 8% judge false-positive rate, a 99% correct parser gets published at 91.1% accuracy.
- Ask for the judge's false-positive rate on tables that are correct by construction, because without it a one-point gap cannot be interpreted.
Every Table OCR Score Passes Through a Judge
A table OCR benchmark score measures two things at once: the parser and the scoring function that grades it.
Deterministic metrics compare the parsed table against a reference in a fixed way. TEDS (Tree-Edit-Distance-based Similarity) turns both tables into trees and counts the edits between them. Cell-level F1 compares values by position. CER (character error rate) counts character insertions, substitutions, and deletions. Each is reproducible, cheap, and built on a normalization decision: whether case matters, whether 1,234.00 equals 1234, whether a blank cell equals an empty string.
An LLM judge replaces that fixed rule with a semantic decision. It receives the table image and the parsed output and answers whether content, structure, and formatting were preserved. That handles legitimate variation string metrics punish, and adds a second error source: the judge can mark a correct output as wrong.
The judge's error rate is part of the benchmark, whether or not the paper reports it.
The pattern is not unique to LLM judges. A first-party comparison of eight receipt engines found CER and field-level F1 ranking the same systems in opposite directions (why CER misleads on document parsing). A metric is a lens with a shape, and most reports describe only one.
How an LLM Judge Turns a Table Into One Number
An LLM judge compresses hundreds of cell-level decisions into one verdict, and each compression step is a place where a correct output can be marked wrong.
Matching comes first. Something has to decide which parsed table corresponds to which reference table. Parsers split one table into two, merge two into one, and reorder tables across a multi-column page. A matching step, often another model call, picks a correspondence. If it picks wrong, the judge scores a correct table against the wrong reference and reports errors that do not exist. The 2026 study that validated LLM judging built this pipeline explicitly (arXiv:2603.18652).
Then the rubric. A typical judge answers three questions per pair: are the cell values right, is the structure right, is the formatting right. Some prompts return three booleans; others ask for a 0 to 10 score. Either way, the judge is deciding what "the same table" means, and that decision is not written anywhere inspectable.
Then aggregation. Three booleans collapse into flagged or not flagged. A 0 to 10 score collapses into a mean across the test set. A 40-row table and a 4-row table contribute one number each. A dropped row, a transposed column, and a total off by a factor of ten all become one unit of lost credit.
Where the False Positives Enter
False positives enter at five specific points, and each one produces a different kind of wrong number.
| Entry point | What the judge does | Effect on the score |
|---|---|---|
| Normalization | Decides whether 1,234.00, 1234, and $1,234 are the same value, and whether case or a blank cell matters | The same pair can pass on one run and fail on the next if the rule is implicit |
| Copyediting the source | Reads an obvious typo as the intended word, then marks the faithful transcription as wrong | A correct cell is recorded as an OCR error that never happened |
| Structure equivalence | Treats a merged header and two split cells as different, or a table split by a page break as two tables | Valid encodings are penalized, and the penalty grows with layout complexity |
| Partial credit | Gives fractional credit, so a wrong total and a wrong footnote cost similar amounts | Severity disappears from the score |
| Verbosity | Reads a longer output with restated headers as more complete | Padding can raise the score without adding correct cells |
The first entry point has a documented solution. OCR-D, the German coordination project for OCR, defines its scoring rules before any comparison: explicit ground-truth transcription levels, Unicode NFC normalization, and a normalized CER (OCR-D quality assurance specification). Its transcription guidelines add that printing errors are not corrected, because fidelity includes them (OCR-D ground truth guidelines). An LLM judge makes both decisions inside a prompt, per call, with no versioned rulebook.
Structure has the same shape. TEDS scores the HTML tree, so two encodings a reader would call the same table can produce different trees. The GriTS paper (2023) notes that partial-credit measures became popular partly because they are less sensitive to ground-truth noise (arXiv:2203.12555). A metric that absorbs ground-truth noise also absorbs real errors, a reasonable trade for research and a dangerous one for a buying decision.
Each of these decisions is defensible on its own. Together they make the score a function of the judge's preferences as much as the parser's output.
The Precision and Recall Tradeoff Belongs to the Judge
Tightening a judge so it catches more corrupted tables raises its false-positive rate, and at low error rates the false-positive rate becomes the number you report.
A judge has two relevant properties. Its true positive rate (TPR) is the share of genuinely corrupted outputs it flags. Its false positive rate (FPR) is the share of correct outputs it flags anyway. In benchmark terms, precision is the share of flagged tables that are actually wrong, and recall is the share of wrong tables that got flagged. If both rates held constant, the reported error rate would be:
reported error rate = FPR + true error rate × (TPR − FPR)
Put in a judge with a 95 percent TPR and an 8 percent FPR. These numbers come from the formula, not from a specific benchmark.
| True error rate | Reported error rate | Reported accuracy |
|---|---|---|
| 1% | 8.9% | 91.1% |
| 5% | 12.4% | 87.6% |
| 20% | 25.4% | 74.6% |
| 40% | 42.8% | 57.2% |
Under those assumptions, a parser that is 99 percent correct gets published at 91.1 percent. The 99 and 95 percent parsers still rank in order, but the gap shrinks from 4 points to 3.5. The absolute number is the judge's noise floor plus a compressed version of the real error rate.
Ordering survives only while the FPR is constant, and there is little reason to assume it is. A dense table gives the judge more cells to object to than a four-cell one, and layout conventions vary by document type, so the rate moves with the document mix. At 5 percent FPR on simple tables and 15 percent on complex ones, a parser that is better on complex tables can lose to one that is better on simple tables. The ranking inverts silently.
The tradeoff is a setting, not a property of the parser. A benchmark tuned to catch every injected error runs at high recall and accepts false alarms. A production gate is usually tuned for precision and accepts misses. A team can improve its benchmark score and still see the same exception rate in production.
Production teams manage this lever already. Confidence thresholds move precision and recall, and the threshold sets the size of the review queue (document automation exception rates). A benchmark that reports one score without its judge's operating point hides the same choice.
One engineer running evals described the loop in an r/mlops thread: the judge is inconsistent across wording changes, "biased toward longer answers," rationalizes outputs the gold set marks wrong, and calibration becomes "stacking prompt rules on top of prompt rules hoping the false positives go down" (r/mlops).
What the Evidence Actually Shows
LLM judges track human judgment better than any string metric, and they agree with each other more than trained humans do, which is the part to treat with care.
The strongest current evidence is a 2026 table-parsing study with more than 1,500 human ratings (arXiv:2603.18652). LLM judges correlated with human scores at r = 0.93, against 0.68 for TEDS and 0.70 for GriTS. With parsers that miss tables entirely excluded, TEDS compressed the rest into 0.66 to 0.88, about 22 percent of its scale, while LLM scores spanned 5.75 to 9.55. Semantic judging recovered differences the deterministic metric flattened.
That study also reported the warning. The leading judges correlated with each other at 0.93 to 0.95, higher than the human annotators did at 0.81 to 0.91. The authors describe a shared LLM prior that should be read as low-noise consensus rather than ground truth. Agreement among judges is not evidence that the judges are right. It can be evidence that they share a blind spot.
The reference a judge is given changes its error rate substantially. A 2026 analysis found that, absent a correct reference, judges agree with experts mainly on questions the judge can answer itself, and that a human-written reference reduces both false positives and false negatives. A slightly wrong reference can be worse than none (arXiv:2503.05061). The MT-Bench study reached a compatible result in 2023: on its math questions, the default judge prompt produced 14 wrong verdicts out of 20, and a reference-guided prompt produced 3 (arXiv:2306.05685). The same study documented position and verbosity biases.
A judge with a reference is comparing. A judge without one is answering the question again and grading its own answer.
A stronger judge rarely fixes a broken evaluation. It changes the bias, not the structure. What works is a reference, a written rubric, and a measured error rate for the judge itself.
How to Audit a Judge Before You Trust Its Number
Ask for the judge's false-positive rate on outputs that are correct by construction before you ask for the parser's score.
Build a clean set where the answer is known
Render the reference HTML or Markdown to an image. The image now comes from the answer, so every pair is correct by definition with no labeling judgment involved.
Measure the false-positive rate on that set
Every pair the judge rejects is an error the judge invented. Report that rate beside every score. It is the benchmark's noise floor, and without it a one-point difference cannot be interpreted.
Inject one error at a time, by category
Change one number, drop one row, transpose one column, drop a header, add emphasis. Measure detection per category. A single aggregate rate hides blind spots, including the rubric categories your documents stress most.
Require the judge to show its evidence
Make each verdict include the cell coordinates and source text the judge compared. A rejection it cannot locate is a preference, not a finding.
Calibrate on humans and version the judge
Score 100 to 200 stratified pairs with two annotators, then report agreement between judge and humans, not just between judges. Record the model, prompt, and normalization rules, and re-audit when any changes.
If you are evaluating vendors rather than judges, apply the same logic to their test set using this accuracy testing protocol (accuracy testing protocol).
For a published figure you cannot rerun, four questions get most of the way there. Which judge, model version, and prompt produced it? What is that judge's false-positive rate on correct outputs? Is the error rate per cell, per field, or per table, and are all errors weighted equally? Is the test set drawn from the document types you actually process?
When an LLM Judge Is the Right Instrument
Use a judge where several outputs are legitimately correct, and a deterministic check where only one answer is.
Good fit: semantic equivalence across output formats, reading order, triage of a regression run, and pairwise comparison. Asking which of two outputs is better is more stable than asking whether one is correct, because the judge has a concrete contrast instead of an absolute standard.
Poor fit: numeric totals without a separate arithmetic check, score differences that decide a purchase, longitudinal tracking across judge versions, and claims that must survive an auditor.
A third option sits between string metrics and judges. olmOCR 2 replaced continuous edit distance with binary unit tests, each asserting one checkable fact about a page, because continuous scores do not always track correctness (arXiv:2510.19817). Unit tests are deterministic and can accept more than one valid encoding. They only check the facts someone wrote, so test design becomes the new judgment call, visible in the test file.
The deterministic side needs the same discipline: eight receipt engines rank differently on cost, CER, and field F1, and none of those rankings is wrong (traditional OCR vs document-parsing VLMs). The mistake is reporting one axis and calling it accuracy.
Whatever instrument you use, publish two numbers: the score and the judge's false-positive rate. A benchmark that reports both makes a claim you can check.
The same standard works on your own files: compare a sample of cells against the source before deciding a score is good enough. A review layer that highlights the source region behind each cell makes that check fast.
Frequently Asked Questions
How do I know if an LLM judge is wrong about my table?
You cannot tell from the score alone. Build a set where the answer is known by construction, measure how often the judge rejects those pairs, and require it to point at the cell and source text behind each rejection.
What is a normal false-positive rate for an LLM judge?
There is no published baseline. The rate depends on the model, the rubric, the document type, and how much formatting the rubric asks about. A judge that flags 5 percent of clean tables and one that flags 20 percent can both be right for different jobs.
Why does a better table parser score lower on a benchmark?
The judge's false-positive rate may be higher on the documents where the parser is strongest. The parser may also use an encoding the reference does not, such as split header cells or Markdown instead of HTML, or the matching step paired it with the wrong table.
Should I use TEDS or an LLM judge for table extraction?
Report both when the decision matters. TEDS is cheap but penalizes equivalent encodings. An LLM judge is closer to human judgment but needs its own audit. Their disagreement tells you which failure mode your case cares about.
Table OCR benchmarks will keep using LLM judges, because the alternative is a string metric that penalizes correct tables for using a different encoding. The useful question is no longer whether to use a judge. It is whether the benchmark measured its judge before it measured the parser.