PlexaraWhy measure at all
Plexara has been built the way durable software usually is: iteratively, with every version proven against real client data before it shipped. That is how we know the context layer works. It is not, on its own, how we can prove it to someone who has not watched it run. Field experience tells you a thing helps. It does not tell you by how much, on which kinds of question, or where the platform is still leaving accuracy on the table. For that you need a number you did not choose.
So we built a benchmark. The goal is not a marketing figure. It is an instrument: a repeatable way to isolate what the platform contributes, separate from what the model contributes, so that we can both confirm the production experience in the lab and see precisely where the next iteration should go. The results below are drawn from controlled runs that cost roughly 158 million tokens, and we publish them the way we intend to keep publishing them, including the parts that are not yet flattering. The rigorous version, with figures, confidence intervals, and reproduction commands, lives in the full benchmark report.
The one thing that changes is the platform
The design decision that makes the number trustworthy is what the benchmark holds still. Every run uses the same model, the same prompt scaffold, the same seed data, and the same task set. The only thing that varies between runs is the platform configuration. We call these configurations arms, and they are ordinary config profiles, not code forks, because the platform's own configuration surface is the thing under test.
The benchmark ladders through four arms, each turning on one more layer. The two that carry the headline are the ends of that ladder. The baseline, A0, gives the agent raw toolkit tools only: it can run queries against the warehouse and read objects from storage, with no semantic enrichment, no search, and no knowledge pages. It is what you get when you wire an AI agent straight to your data with a stack of connectors. The shipped platform adds semantic cross-enrichment on every result, a search tool for discovery, and the curated knowledge pages that carry an organization's business rules.
Because the model, the questions, and the data are identical across arms, any difference in the answers is attributable to the platform and nothing else. This is the opposite of the usual AI demo, where a better result could always be a better model, a luckier prompt, or a friendlier question. Here the model is held constant on purpose. If the platform arm wins, the context layer is why.
Ground truth you cannot argue with
A benchmark is only as honest as its answer key. Ours is generated, not written. One fixed-seed dataset model produces everything from a single source: the warehouse tables and their rows, the catalog metadata and column descriptions, the knowledge pages, and the task questions. The correct answer to each question is computed from the generated rows, never typed in by hand, so there is no opportunity to quietly tune the key to the result. A determinism test fails the build if the committed artifacts ever drift from what the generator produces.
Measurement runs through the platform's own audit log rather than by reading the transcript and guessing. The harness threads a session handle invisible to the agent and reads efficiency metrics back from the admin audit API. A run fails loudly when a session's audit rows do not reconcile against the harness's own accounting of what it called, and attempts that fail at the harness level, a dropped connection or an audit read-back error, are excluded from accuracy and reported separately. A harness bug is never allowed to count as a wrong answer, and correctness with repeats is scored as pass-of-k: every one of the three attempts must be graded and correct.
The run covers three suites at three repeats each, 261 graded attempts per arm across four arms, with zero harness failures. S1 is discovery: which table answers this question. S2 is analytical accuracy: exact numeric answers, some of them SQL graded by executing the query. S3 is knowledge traps: questions that have a plausible wrong answer you will reach unless you know a business rule that lives in the semantic layer.
Where the platform earns its place
On discovery and arithmetic, the arms are effectively tied. A capable model finds tables and computes sums without help, so S1 and S2 sit near the ceiling for every configuration. The platform does not make the agent better at questions it could already answer, and on the easy suites its search-first step even costs a few extra tool calls for the same result. That is worth stating plainly, because it is the shape of an honest instrument: the context layer is not a universal accelerant, and pretending otherwise would be the kind of claim this benchmark exists to prevent.
The knowledge traps are where the arms separate. The raw-tools baseline answered 42.7 percent of them correctly. The full platform answered 98.7 percent, a gain of 56.0 points (95 percent confidence interval, plus 44 to plus 67), and it did so in fewer tool calls: a median of 10 against 16. The platform was both more correct and more efficient, on the same questions, with the same model. The trap-class breakdown shows the mechanism, and it is legible. Some facts, like amounts stored in cents, live in the catalog's column and dataset descriptions, so cross-enrichment alone recovers them. Others, like a February fiscal year or the definition of a key account, live only in knowledge pages, so bare tools and enrichment both score zero on them and only search recovers them. Each trap is defeated exactly when the channel that carries its fact is switched on.
The cleanest single example is a question about net revenue. Revenue in this dataset is a policy, not a column: it is the order amount minus the discount, counted over completed orders only, and that rule lives in the dataset description and the revenue-policy knowledge page. Without the rule, an agent returns the gross total, a confident, professional, wrong number. On this trap class the raw-tools baseline scored 13 percent; the platform scored 100. The baseline was not broken. Its queries ran and its arithmetic was fine. It was working without the one fact that changes the answer, and it had no way to know the fact existed.
That is the whole thesis of the platform, now with a measurement attached to it. More tools do not make an agent smarter; the missing piece is almost never a tool, it is knowledge about the data the tools touch. The benchmark turns that argument from a position into a delta you can reproduce.
What this is, and is not
We report these numbers with their limits attached, because the limits are the point of measuring. The headline is arm-vs-arm on a single pinned model; it is the platform's effect, not a claim that one model beats another. The seed dataset is small and airgapped by design, so the absolute accuracies are not real-world estimates. What the ablation isolates, the platform's contribution holding everything else constant, is what the exercise is for.
A second suite measures the memory-to-knowledge lifecycle: can a fact taught in one session be captured, promoted, and reach a different user later? Here the honest reading is more mixed. The platform reliably surfaces a saved memory once captured, and it never fabricated a fact it was not taught, but cross-user transfer sits near 45 percent across repeated runs, and several of the finer metrics carry enough sampling noise at this scale that we report them as ranges rather than point estimates. That capture-and-spread loop is maturing, not finished, and we say so rather than market it. The reason to run a benchmark you might lose is that the losses are the roadmap.
The harness is open. The arm configurations, the deterministic seed generator, the graders, and the audit-derived metrics all live in the benchmark module of the open platform, and the full report carries the figures, confidence intervals, and the exact commands to reproduce every number. That is the standard we want to be held to: not that Plexara makes agents better, but that you can check.