75%→0%

The apparatus is a deterministic HTTP service behind the platform’s API gateway, serving a social-analytics-shaped catalog: monitors, trend series, profile metrics, a workspace structure. A harness-only control plane changes the account’s world between sessions, so a belief planted in one world can be queried in another, and the service’s access log spans the change: a recheck after the world moved is detectable as a decision, not inferred from the transcript. The cost of re-establishing the state is a world property, from one call on an unscoped account to eleven on a workspace-scoped one.

Beliefs are frozen, committed strings planted through the platform’s real capture tool, as the same identity that will later be asked. A cell pairs a question, a belief (or none), and a query world; its correct behavior is derived from two computed facts (is the question answerable in this world; is the belief true in it), never hand-assigned. Grading is deterministic: verification is read off the fixture access log under a definition fixed before any data, refusals key on a prescribed sentinel, and numeric answers grade against ground truth computed from the same data the service serves.

Episodes run as fresh platform identities over MCP through two drivers: the Claude Code client, and an in-process tool loop against the raw model API with no agent client, added specifically to strip client behavior from the headline results. Models are Sonnet 5 (both drivers) and Haiku 4.5, at k = 8 per cell with Wilson 95% intervals. Every headline table was rerun on a platform binary built from release tag v1.116.0 and replicated exactly; every run archives its manifest, per-attempt records, and full transcripts. All findings are exploratory, with decision rules stated before each run, and the report says so plainly.

Delivered a true note about checkable world state, Sonnet 5 re-checked it against the live service in every attempt, and no lever we pulled changed that. Sweeping the price of checking from one call to eleven moved nothing: verification held at 8/8 at every cost level, and the agent paid the full price rather than sampling, enumerating all ten workspaces at the top of the range.

Even a note that states the answer outright changes nothing. With the note, Sonnet 5 verified 32/32 and trusted 0/32; matched no-knowledge controls verified 16/16; the median effort difference between being handed the answer and being handed nothing was exactly zero calls. The note is not missed: it appears in the transcripts and is cited in most final answers, as corroboration of a result the agent had already re-derived. The raw-API runs, with no agent client in the loop, reproduce this exactly, 48/48 verified.

For a buyer evaluating a knowledge layer, this null result is load-bearing good news: on a capable model, delivered knowledge never becomes blind trust. The agent treats the platform’s notes about the current state of the world the way a good analyst treats a colleague’s recollection, worth citing, worth confirming, never a substitute for looking.

The null above admits a deflationary reading: perhaps this model ignores delivered notes entirely. The bridge experiment tests that with a belief whose content cannot be re-derived from any endpoint: an internal reporting convention stating that a monitor day counts as positive coverage when its sentiment score is 70 or higher, paired with a question that requires real analytical work plus the convention. Thresholds from 50 through 80 all yield distinct day counts, so any stated answer betrays the threshold that produced it, and the no-note control doubles as a leakage check.

With the note, Sonnet 5 used it in 8 of 8 attempts: every attempt fetched the trend series and counted days at threshold 70. Without it, no control attempt ever produced the note-only answer (zero leakage), two declined, and six fabricated a definition, adopting a threshold of 50 as a “neutral midpoint” and answering confidently. A delivered convention therefore does double duty: it is used, and it suppresses confident invention of institutional facts, 75% fabrication without it, zero with it.

This is the study’s center of gravity. Conventions, definitions, policies, fiscal calendars, deprecations: facts with no endpoint to consult are exactly the facts an agent cannot recover on its own, and exactly the facts it will invent plausibly when missing. They are also exactly what Plexara’s memory-to-knowledge lifecycle captures from real sessions, curates, and delivers. The weak tier agrees: Haiku 4.5 used the convention in 5 of 6 clean attempts, and its controls fabricated in 6 of 6.

On the identical cells, the weaker tier inverts the world-state result: Haiku 4.5 trusted the delivered answer-bearing note in 29 of 32 attempts, verifying in 3, while its no-knowledge controls probed 16 of 16. The trust is shaped rather than blanket. A stale note that offers no answer to adopt still gets probed; the exposure is answer-bearing notes, and the stale-answer cell prices it directly. The note says three monitors, the account has been emptied, the truthful answer is zero:

Measured against its own perfect no-note control, the stale note took Haiku 4.5 from 8/8 to 0/8. The strong tier’s re-derivation habit makes it staleness-immune on the same cell, 8/8 in both conditions. The deployment guidance falls straight out of the data, and we give it to customers as measured fact rather than folklore: delivered conventions and definitions are safe and valuable on every tier tested; delivered observations of world state pair best with capable models, which treat them as corroboration and re-verify them for free. That is also how we run Plexara in production, with frontier-class models doing the analytical work.

The results form a two-factor structure, derivability crossed with model tier:

Read together with the accuracy study, the picture is consistent, and each study independently explains the other. The accuracy study’s +56-point knowledge-trap lift came precisely from non-derivable facts: unit conventions, net-revenue policy, fiscal calendars, deprecations. This study shows why those facts and not others carry the lift: they are the knowledge an agent relies on completely because no query can reconstruct them, and the knowledge it fabricates most confidently when missing. The value of a knowledge layer concentrates in what agents cannot re-derive. Two studies, two methods, two model tiers, one conclusion.

The findings also drove platform decisions, in both directions. Retired, with the evidence above: volatility and valid-until schema fields, freshness-and-recheck-cost enrichment, and capture steering toward dated observations, features the falsified hypothesis would have justified. Sharpened, with new evidence: capture filing, where a companion lifecycle probe found the supersede mechanism at ceiling and located the real loss stages upstream in entity linking, and strict input validation at tool boundaries, surfaced by an agent burning calls on a silently ignored misnamed field. The benchmark series is how we decide what to build: the numbers retire features as readily as they justify them.

Every table in the published report regenerates from raw run data committed to the open platform repository, offline, with no API key and no network access. Run manifests pin the commit, platform build, model, driver, client version, seed-set hash, and k. The release-tagged rerun means the headline numbers are tied to a build anyone can produce from the tag.

git clone https://github.com/txn2/mcp-data-platform.git
cd mcp-data-platform
python3 bench/reports/knowledge-use/pk_tables.py   # every table, from committed raw data
python3 bench/reports/knowledge-use/figures.py     # every figure

The report and a snapshot of the raw run data are archived on Zenodo under DOI 10.5281/zenodo.21614059, with a frozen PDF as part of the archive.

• Two models, one family, one pair per tier: the capability axis is a two-point contrast, and "tier" is confounded with everything else that differs between Sonnet 5 and Haiku 4.5. The claims are about these models, not about capability in general.
• The headline cells rest on a handful of questions over one fixture family. The derivability contrast is controlled within that fixture; breadth is the price of the controls.
• The reporting convention in the bridge is study-authored and declared as such; its leakage control is structural (distinct answers per threshold), but its phrasing is ours.
• Every finding is exploratory. Decision rules were fixed before each run, but hypotheses were revised between runs as premises fell, so intervals are per-rate Wilson bounds, not a corrected confirmatory family.
• The raw-API replication covers the strong-tier headlines but not Haiku, whose results are client-path only; four Haiku bridge episodes were lost to API errors and are excluded as failures, not graded.
• The release-tagged rerun replicated every headline; two secondary rates moved within noise and toward the conclusions (control fabrication 6/8 to 8/8, weak-tier outright trust of the stale note 6/8 to 8/8).

1. [1]Johnston, C. (2026). Does a Semantic Knowledge Layer Make an Agent Measurably Better? A Reproducible Benchmark. mcp-data-platform benchmark report series. link
2. [2]Zep: temporal validity ledgers (valid-at / expired-at / invalid-at edges) for agent memory graphs. link
3. [3]MemStrata: deterministic supersession of agent memory by subject-relation-object. link
4. [4]FAMA: forgetting-aware memory accuracy, penalizing reliance on obsolete memory. link
5. [5]STALE: staleness detection in purely conversational agent memory, where no verification action exists. link
6. [6]DRNoise: falsifiable misleading evidence for document-research agents, without an operational world state. link