What one person teaches, the whole team gets

The measurement is a scripted protocol rather than an impression. One session, under one identity, is taught a fact the data cannot reveal on its own: a definition, a fiscal boundary, a convention about how a column is stored. A reviewer then applies that capture, which is the step that turns one person’s note into something the organization asserts. Then a fresh session opens under a completely different identity, with no access to the first conversation, no hint that anything was taught, and no pointer to the person who taught it, and it is asked a question that cannot be answered correctly without the fact.

Grading is on the answer, not on effort. Either the platform put the fact in front of the second identity in time to be used, or the answer is wrong. The suite runs 30 of these protocols end to end on claude-sonnet-5, repeated as 5 independent passes, which leaves 149 protocol-runs after one exclusion for a harness fault.

It is worth being clear about what this does not measure. It says nothing about whether the fact was worth teaching, whether the reviewer should have applied it, or whether the second person would have thought to ask. It measures one link in the chain: a promoted fact reaching a colleague who was never told about it.

On build v1.102.0, across 15 protocols at k = 3, a fact promoted to shared knowledge was reused correctly by a different identity 46.7 percent of the time. Under half. Every other stage of the lifecycle in that same run looked healthy: the agent captured what it was taught, a later session under the same identity recalled it, search surfaced it without being asked. The link that failed was the one the feature exists for.

We published that on the front of the report as a limitation rather than burying it in a threats-to-validity note, and we published a reading of the cause with it: the gap looked more like a capture-and-propagation limit than a surfacing failure. In plainer terms, the guess was that the fact was not becoming visible to the second identity in the first place, rather than being visible and getting ignored.

Publishing a failing number on a capability you are selling is uncomfortable, and it is also the only way the number is worth anything. A benchmark that reports only its wins tells a reader nothing about what the instrument would do if the product were broken, because it has never been observed doing anything else.

The change that followed was small and specific. Visibility moved to the act of applying an insight: the moment a reviewer applies a capture, it becomes findable by every identity through the same search everyone already uses, still attributed to whoever taught it. Before that, a fact filed against a catalog entry rather than onto a knowledge page could reach nobody but its author unless a question happened to name the exact table it hung off. That is the precise path the earlier reading measured.

The suite was then re-run at k = 5 on build v1.118.0, with 30 protocols instead of 15. A promoted fact reached a different identity and produced the correct answer 98.9 percent of the time, 94 of 95 transfer attempts, with a 95 percent bootstrap confidence interval of 96.8 to 100.0. The interval touches the ceiling, which is a way of saying the run found one failure and cannot rule out that a longer run would find a few more.

The part worth more than the number is that the earlier reading of the cause held up. The fix was aimed at the mechanism the report had guessed at, and the metric that moved was the one that mechanism controls. A benchmark that only produces a score tells you where you are. A benchmark that produces a diagnosis you can act on, and then confirms or refutes it, is an instrument you can steer with.

Those two figures come from different platform generations, and the report says so in the section header rather than the footnotes. The ablation and cold-start studies are pinned to v1.102.x; the lifecycle suite was re-run on v1.118.0. Report 2.0.1 states that the sections are not mutually comparable and that no number should be carried across the boundary.

For the reach pair, crossing that boundary is the entire point. A change landed between the two builds, the pair is what the change did, and nothing else about it would be measurable if both readings came from the same code. Sample size moved too, from 15 protocols to 30 and from k = 3 to k = 5, so the honest statement is that a targeted change and a larger run happened together, and the report argues the mechanism rather than resting on the size of the jump.

For every other metric in that section, the line holds. Anything else that differs between the two readings differs for reasons nobody isolated, and reading a trend into those pairs would be reading noise.

A near-ceiling accuracy number invites a fair objection: maybe the second identity worked the answer out on its own and the knowledge layer was decoration. Report 2.0.1 added a decomposition to answer it. The taught fact was provably delivered to the second identity in 68.4 percent of attempts [58.9 to 77.9], and in 98.5 percent of those the answer that followed was correct.

The gap between 68.4 percent delivered and 98.9 percent correct is the check being deliberately hard on itself. Delivery counts only when the stored fact appears as a literal normalized substring of a tool result, which means the check can miss a real delivery that got paraphrased, but it cannot manufacture one. All 30 of the correct-but-not-counted episodes were read back against their transcripts, and in every one the agent had searched and a tool result carried the protocol’s key term. None of them is a case of the model deriving the answer unaided.

The other half of the decomposition is the more interesting half. Once a fact is actually in front of a capable model, it is used 98.5 percent of the time. Getting the fact there is the hard part, and it is the part a platform is responsible for. That matches what our knowledge-use study found from the other direction: strong models re-verify what they can check and lean completely on the conventions and definitions they cannot re-derive.

The reason to run a benchmark against your own product is not the marketing number at the end. It is that a suite instrumented stage by stage tells you which stage broke. The reach failure showed up as a single bad row next to seven healthy ones, which is what made it diagnosable at all, and the same instrumentation is what verified the fix rather than a set of hand-picked demos.

That is also the practical argument for owning the loop instead of waiting for a better model, which we made at more length here. The knowledge your business runs on lives in your people, and the mechanism that moves it from one person to the next is something you can inspect, measure, and repair. A model upgrade will not do it for you, because the missing fact was never in the model.

Concretely, for a team on Plexara: the hour an analyst spends untangling a definition is an hour that is spent once. The next person to ask, in another department, in another quarter, with no idea the first conversation happened, gets that work handed to them. The method, the confidence intervals, and the reproduction commands are in the full accuracy study, the rest of the research sits in the research center, and what the capability looks like in the product is on the knowledge application page.