96% vs 0%

Three completion cells, each a document an operations team actually writes: a schema change plan for a governed data stream, an incident-handling write-up, and the onboarding of a new nightly data feed. Each cell’s governing constraints live on 5 to 8 pages spread through an operations wiki, up to four references deep from the entry page the agent is handed. Every constraint carries a hard token that exists only on its source page, so grading is deterministic: a constraint counts as grounded only when its token is in the final document and the source page was actually fetched during the episode. A token in the document with no read behind it is counted separately, as fabrication.

Two corpora with the same prose meaning: a connected arm, where pages reference each other the way Plexara knowledge pages do, and a stripped arm, where each reference is rendered as its plain-prose fallback sentence instead of a link. Crossed with search on or off, and run at three corpus sizes, 50, 500 and 5,000 pages, generated deterministically from a recorded spec so any reader can regenerate the exact corpus. The three cells and their 27 core pages are byte-identical at every size; only the haystack grows. The matrix, the dependent variables and the kill conditions were frozen in a design document before any episode ran.

The premise probe ran 72 episodes on a 42-page corpus at two reading budgets. The confirmatory matrix ran 99 pre-registered episodes across the three sizes with one agent configuration, moving the ratio of reading budget to corpus size two orders of magnitude. One episode died to a harness error and is archived as such, not graded.

Hand an agent one entry page, no search tool, and a task whose constraints sit up to four references away, and it walks. On the large reading budget the agent recovered 96% of the spread constraints by pure reference-following, at zero queries, reaching the deepest page of every cell in all nine episodes. The small budget recovered 42%, and its failures were stopping early, not failing to start. On the stripped corpus the same agents recovered nothing, because nothing gave them a route.

The walk survives scale. In the confirmatory matrix’s no-search condition at 5,000 pages, every episode was a full-depth pure-reference walk: 9 reads per episode, which is the size of the reference closure itself, every constraint grounded, all nine episodes. Voluntary traversal is not an artifact of a small corpus.

With search on, both corpora reach the same place: grounded coverage is at ceiling in every cell of the matrix. At 5,000 pages an episode makes about 11 reads, roughly 0.2% of the corpus, and still grounds every constraint. Discovery is not enumeration; targeted search plus the vocabulary an agent picks up from the pages it reads collapses a 5,000-page haystack as effectively as a 50-page one. What scale moves is the cost.

On the connected corpus the effort curve is flat: 0.59, 0.67, 0.63 searches per grounded constraint across two orders of magnitude. Stripped, it climbs: 0.73, 1.02, 1.44, plus the only failed episode and the only below-ceiling coverage in the matrix, both in the 5,000-page cell. And agents lean on the references more as the haystack grows: the share of reads discovered through a reference on a fetched page rises from 0.09 at 50 pages to 0.34 at 5,000, while the stripped arm stays pinned to search for every single read.

The 50-page cells are the built-in control: at a size where a raised search limit can enumerate the corpus, the design predicted no separation, and both arms sat at identical ceiling with a coverage delta of exactly zero. The separation that appears at scale is a scale effect, not a haystack artifact.

The probe’s small-budget arm is where references stop being an efficiency and start being the difference between a grounded document and a thin one. With search on, the connected corpus nearly tripled grounded coverage, 0.29 against 0.10, and cut the searches spent per grounded constraint from 8.0 to 2.3. The characteristic small-budget failure was searching without reading: in two thirds of its search episodes the agent issued queries, fetched nothing, and wrote the document from result snippets.

The durable variable is not the model, it is the ratio of reading budget to corpus size. The probe’s large budget could afford to read half of a 42-page corpus and hit ceiling with or without references; nothing can read half of a real knowledge base. As the corpus grows, every agent becomes the constrained one, which is exactly the regime the confirmatory matrix measured, and where the cost separation lives.

The stripped no-search arm has a floor of zero by construction: the off-entry pages are unreachable. The large-budget agent still produced constraint tokens for 19% of the slots it had no way to read, 13 slots across nine episodes, stating plausible values inside otherwise honest documents.

This is the completion-task face of the result the knowledge-use study measured on questions: starve an agent of a source and a confident value appears anyway. It is why every headline on this page is grounded coverage, token in the document and source page read, and why read provenance, not text overlap, is the only defensible way to grade a completion instrument.

The confirmatory design pre-registered a stronger construct than cost. Six constraints were authored as semantic discontinuities, institutional obligations like a finance close calendar governing when a schema change may ship, written entirely in their own department’s vocabulary, never naming the task’s systems. Twice per corpus size, an offline embedding scan and a live sweep gate certified that no task-derived query ranked those pages. The claim under test: an authored reference crosses in one hop a gap that search cannot cross at all.

The stripped arms grounded those constraints anyway, at 1.00 coverage at 500 pages and 0.93 at 5,000. The design names any stripped-arm discontinuity grounding an instrument kill, so the kill fired and no confirmatory conclusion is read from those cells. What happened is worth more than the construct: the stripped corpus renders each reference as a meaning-preserving prose sentence, the agent reads that sentence on an ordinary page it found through search, and then searches again in the institution’s own vocabulary. It runs the traversal in query space, two hops instead of one.

The certifications were sound for what they measured; they measured queries derived from the task, and the defeating queries were derived from corpus text the agent had already read. Prose that preserves a page’s meaning necessarily names the institution, and naming the institution hands a competent searcher the vocabulary that closes the gap. An arm contrast that removed the mention would change what the page says and grade two different documents, which the design refused by construction. So the construct is not just unmeasured, it is unauthorable under a meaning-constant contrast, and we retired it rather than weakening the contrast to save it.

• References do real work. Agents use them voluntarily, lean on them more as the knowledge base grows, and depend on them entirely when search is off or the reading budget is small. Connecting a knowledge page to the pages that govern it is not decoration; it is what keeps discovery affordable at the sizes real knowledge bases reach.
• The claim we make is the measured one. For an agent that searches competently, references did not change what the finished document contained at any tested size; they changed what it cost and how it degraded. We say that, rather than the grander completeness claim the kill took off the table.
• Grounded coverage is the reading that counts. The fabrication channel is real and measurable, so any completion claim, ours or a vendor’s, should be graded on read provenance, not on whether the right words appear in the output.
• What we did not learn. Whether agents know when they are done is unmeasured: no episode in 98 ever claimed completeness, so the overclaim channel never separated anything. And whether some constraint can be truly unreachable to read-informed search is a question our instrument cannot pose, for the reason the kill section states.

Both run families are committed whole to the open platform repository: every manifest, graded attempt, transcript, gate reading and plant record, beside the frozen design document. Each family carries a stdlib-only analyzer that recomputes every number on this page offline, with no API key and no network. The corpora themselves are not stored; each archive records a spec and fingerprint from which the exact corpus regenerates, and the harness refuses on mismatch.

git clone https://github.com/txn2/mcp-data-platform.git
cd mcp-data-platform
python3 bench/reports/graph-completion/graph_tables.py   # every table
python3 bench/reports/graph-completion/figures.py        # every figure

The recompute is a build gate rather than a convenience: the project’s own verify step re-derives the headline numbers from the archives and fails on any drift from what the report prints, including the presence of the instrument kill and the archived analyzer’s deliberate non-zero exit. The analyzer exits non-zero by design: the pre-registered kill is present in the archives, and it refuses to bless kill-affected cells as confirmatory findings. That refusal is itself part of the published record, and this page reports those cells the same way. The report and a snapshot of the raw run data are archived on Zenodo under DOI 10.5281/zenodo.21881798, with a frozen PDF as part of the archive.

• One agent client, and model aliases rather than resolved model identifiers: the archives record what the client was asked for (its small and large aliases), not a pinned model build. Budget-to-corpus ratio, not model identity, is the variable the design treats as durable.
• One authored corpus genre. Three hand-written completion cells inside a generated operations wiki, deterministic by construction; other document kinds and messier corpora are untested.
• Fifteen episodes per confirmatory cell resolves ceiling against collapse and a doubling of cost; it cannot resolve small differences, and nothing here claims finer resolution.
• The completeness-closure question is unmeasured, not answered: no episode ever claimed its document was complete, so the overclaim reading never separated the arms in either direction.
• The semantic-discontinuity cells are invalid by the pre-registered instrument kill and are reported as such; no conclusion about unreachable-to-search constraints is drawn anywhere on this page.
• The largest tested corpus is 5,000 pages. The cost curves justify no extrapolation beyond it.
• The tasks are posed and the corpus is authored; production agents on production knowledge bases are a separate instrument, not this one.
• Runs are pinned to recorded development commits of the platform, not a tagged release build, and the archives record the exact commit of each run.

1. [1]Johnston, C. (2026). Does a Semantic Knowledge Layer Make an Agent Measurably Better? A Reproducible Benchmark. mcp-data-platform benchmark report series. link
2. [2]Johnston, C. (2026). When Do Agents Use Stored Knowledge? Derivability, Capability, and the Limits of a Knowledge Layer. mcp-data-platform benchmark report series. link
3. [3]Johnston, C. (2026). Knowledge Pollution: Verification Displacement, Capability, and the Price of a Curation Gate. mcp-data-platform benchmark report series. link
4. [4]Graph-structured retrieval for query-focused summarization: builds the graph from the corpus rather than measuring authored references, and evaluates answer quality rather than grounded completeness. link
5. [5]Knowledge-graph-indexed retrieval motivated by long-term memory consolidation; the retrieval structure is derived, and single-answer recall is the metric rather than document completeness. link