A wrong number is usually a scope error, not an arithmetic one

I reported a recall of 0.699. It was 0.641. Nothing had been miscalculated.

The arithmetic was correct throughout. Every step of the computation did exactly what it was written to do. The figure was wrong because the comparison was drawn across the wrong population, and none of the checks I had could see it.

By the time I caught it, 0.699 had left the script that produced it and entered session notes, a weekly slide deck and an executive summary. Nearly six points, in this case, was the difference between “extraction is good enough to scale” and “extraction needs another prompt iteration before we commit GPU time to a hundred thousand articles”. We were one meeting away from the expensive version of that decision.

The bug was a denominator, not a calculation

The pipeline extracts subject–predicate–object triples from biomedical papers. Evaluation compares what was extracted against a hand-curated ground truth.

My scoring script flattened the extracted triples from all three evaluation papers into a single list before matching. Ground truth stayed correctly partitioned by paper. The predictions did not. So a triple extracted from paper A could be matched against a ground truth triple curated from paper B. Where both papers discussed the same intervention and outcome, which in a corpus on diet and metabolic disease is most of them, it scored as a hit.

It is not a hit. Extraction is a per-document task. A model that reads paper A and produces a relationship attested only in paper B has not extracted anything; it has recovered a base rate of the domain. The correct denominator was never the evaluation set. It was always the paper.

Constraining the match per-paper took true positives from 109 to 100 and recall from 0.699 to 0.641. Precision did not move at all, because the extracted count never changed, only what those triples were permitted to match against. That asymmetry is precisely why it survived: half the metrics looked exactly as they should.

The miss categorisation shifted the same way, and more instructively. The share of misses where both entities had been extracted but not correctly paired went from 74% to 77%. Reassuringly stable. The absolute count went from 35 to 43. There was 23% more recoverable work sitting in that category than I had told anyone, and the percentage, the number we had been quoting, concealed it completely.

The figure survived because it was mildly encouraging

Every check I had was downstream of the bug, so none of them could see it.

It was also plausible. It sat close to the previous prompt iteration's result and slightly above it, which is exactly the shape of finding that attracts no scrutiny. A figure that contradicts a prior gets re-run. A figure that mildly confirms one gets copied into the next document.

That copying is the more consequential failure. By the time 0.699 reached the slide deck it had been transcribed twice, and at each transcription it shed the context that would have let anyone check it: which ground truth version, which prompt version, which scoring regime, per-paper or aggregate matching. It arrived at the meeting stripped of everything that made it falsifiable. It had stopped being a measurement and become a number.

An independent script in the same project, deep_diagnosis.py, computed overlapping quantities and would have disagreed. Nobody ran both. No rule said to.

Five requirements now govern any figure entering a document

The script fix was one line. The fix that mattered was to what a figure must carry before it is allowed into a document at all.

1. Script attribution. Every number in a table, slide or summary names the script and function that produced it, in the document: # Source: eval-v3/rescore_pair_detail.py → pair_analysis(). Derived figures show the derivation.

2. Scope declaration. Every ratio states its denominator. “74% of missed pairs” is not a statement. It becomes one when it specifies which pairs, under which ground truth version, which prompt version, and which matching regime. This is the rule that would have caught the bug, and it exists because it didn't.

3. Re-derivation before propagation. A number may not be copied from a session note into a slide. It is re-derived from source and confirmed. Stale figures are treated as unverified figures, which is what they are.

4. Cross-checking against an independent computation. Where two scripts compute overlapping metrics, they are reconciled before either is presented. Where they disagree, the reason is stated (differing unit, threshold or scope) along with which is being used and why.

5. No ungrounded figures. If you cannot point at the line of output that produced a number, the number does not go in. “Approximately 74%, from memory” is not a finding.

Make the unverified claim harder to write than the verified one

The instinctive response to an error like this is a commitment to greater care. That is not a control. It is a disposition, and dispositions decay under deadline.

What holds is structural. Once a slide template carries a source line beneath every table, filling it with a script path is easier than leaving it blank and hoping nobody asks. The discipline stops depending on whether anyone happens to feel rigorous on a Thursday afternoon.

This matters more, not less, as pipelines accelerate. We screen roughly 2.9 million candidate records and have extracted over a million triples. The volume of numbers a project like this generates has risen by orders of magnitude; the human capacity to sanity-check them has not moved at all. Provenance is the only property that scales.

One recommendation to anyone building a comparable evaluation harness: adopt scope declaration first, before script attribution. Attribution tells you where a number came from, which is valuable once you already suspect it. Declaring the denominator forces the author to say what is being measured, which is what catches a matching-scope error while it is still cheap.

Our data was already held to this standard. Every edge in the knowledge graph carries its source PMIDs and the exact sentence it derived from, so any assertion in the graph traces back to the text that produced it. It took an inflated recall figure to show that we were holding our measurements to a lower evidential standard than the thing they were measuring. That is the wrong way round, in a project whose entire output is provenance.

The extraction pipeline described here is built for the Koslicki Lab at Pennsylvania State University: LLM-based triple extraction from biomedical literature into a provenance-preserving Neo4j knowledge graph, with a graph neural network ranking interventions by evidence quality.

← Home