Skip to main content
Field notes / product

When do agents use what you teach them?

Plexara's knowledge-use study asks what an agent actually does with delivered knowledge. Strong models re-verify anything they can check, and rely completely on the facts they cannot re-derive: conventions, definitions, policies. Without those facts, they invent plausible substitutes 75 percent of the time.

8-minute readProduct

The question underneath the first number

Our first benchmark study produced a number we lead with often: on questions that turn on a business rule, an agent on raw data tools was right 42.7 percent of the time, and the same agent on the platform was right 98.7 percent. That study measured whether the knowledge layer delivers. It left a sharper question open, and a careful buyer eventually asks it: delivery is not use. The platform can place a fact in front of an agent, but whether the agent acts on it, double-checks it, or quietly ignores it is a fact about model behavior, not about platform engineering.

We could have answered that question the way most of the industry does, with intuition and a demo. We answered it the way we answer everything about model behavior: we never assume what a model does, we measure it. The result is the knowledge-use study, the second entry in our research series, and its findings changed what we build.

A study that killed its own premise

The study began as a pre-registered benchmark of a hypothesis we found plausible: that agents under-verify stored beliefs that can go stale, and that attaching freshness metadata, how volatile a fact is, how old the observation is, what a recheck costs, would fix the deficit. The protocol fixed the hypotheses, decision rules, and falsifiers before any data existed.

The hypothesis died on first contact with the data. Given a stored belief it could check against the live world, the strong model checked it essentially every time, no matter what checking cost. We swept the price of re-verifying from one API call to eleven; verification did not move. We handed the agent a note containing the literal answer; it re-derived the answer anyway and cited the note as corroboration. There was no under-verification deficit to fix, so the freshness features the hypothesis would have justified were retired before a line of product code was written, and the falsification is recorded in the public protocol, with the runs that killed it archived beside the runs that succeeded.

We tell this part of the story first because it is the part that makes the rest believable. A benchmark that can only confirm what its author hoped is marketing with error bars. Ours is allowed to lose, and when it loses we publish that and change course. The losses are the roadmap, and sometimes they are also the finding.

What emerged instead: derivability

What the wreckage revealed is a clean two-part structure. The first part: whether an agent uses delivered knowledge depends on whether it could re-derive that knowledge itself. A fact about the current state of the world, how many monitors are provisioned, what a metric reads today, is something a connected agent can always go look up, and the strong model always did, making delivery of such facts harmless but redundant.

Then we delivered a fact no query can reconstruct: an internal reporting convention, the kind of rule every organization has, stating that a day counts as positive coverage when its sentiment score is 70 or higher. The experiment was built so any answer betrays the threshold that produced it. With the convention delivered, the agent used it in every single attempt, eight of eight. Without it, the agent did not stop and say it could not know. In six of eight control attempts it adopted a threshold of its own invention, called it a neutral midpoint, and answered confidently. No control attempt ever guessed the real rule.

That is the two-sided result at the center of the study: delivered institutional knowledge is used completely, and its absence does not produce ignorance, it produces confident fabrication. Seventy-five percent of the time without the rule; zero with it.

What this says you should teach your platform

The practical reading is a priority list for what belongs in a knowledge layer. The facts that pay are the ones your agent cannot re-derive from any endpoint: how your fiscal year runs, what net revenue means in your house, which threshold defines a key account, which table is deprecated despite looking authoritative. These are also exactly the facts our accuracy study found carrying the entire +56-point lift, so the two studies corroborate each other from independent directions: one measured how much the knowledge layer helps, the other explains why, and both point at the same class of facts.

This is what Plexara's memory-to-knowledge lifecycle is built to capture. Conventions, definitions, and policies surface naturally in real working sessions, get captured as memories, curated into insights, and promoted into the shared knowledge that enrichment and search deliver to every later session. The study says the agent on the receiving end will use what arrives through that pipe, precisely because it cannot reconstruct it any other way.

And the strong model re-verifying everything checkable is not a disappointment, it is the property you want. Delivered knowledge never became blind trust in any strong-tier cell of the study. The agent treats notes about the changeable world as leads to confirm, not answers to repeat, which is exactly the behavior you would ask of a careful analyst.

Model tier changes the math, and we say by how much

The second part of the structure is capability. A smaller model, run through the identical cells, inverted the world-state result: it trusted the delivered answer-bearing note 29 times out of 32 rather than re-checking. That efficiency has a price, and the study measured it directly with a note gone stale: the small model reported the stale value every time, scoring zero against its own perfect no-note control. The strong model, re-deriving as always, was unaffected.

Delivered conventions were used and fabrication-suppressing on both tiers, so the knowledge that matters most is safe everywhere. The tier result is deployment guidance we can hand a customer as measured fact: pair the knowledge layer with capable models for questions about the live state of the world, and let the layer carry your definitions and policies for every model. It is also how we run Plexara in production, with frontier-class models doing the analytical work.

The full report, with the apparatus, the run directories, every interval, and the honest limitations, is on the study page, and the brand-neutral version is archived with its raw data under a citable DOI. Like the accuracy study, every table regenerates offline from committed run data. Run the scripts and every number in this article reproduces.