Two benchmarks, one conclusion: the industry is converging on context

The dbt study is compact and clean about its method. Eleven business questions run over the ACME Insurance dataset, a schema dbt borrowed from Juan Sequeda and colleagues at data.world, with each question executed twenty times per model to average out run-to-run variance. Four current models were tested: Claude Sonnet 4.6, Claude Opus 4.6, GPT-5.3 Codex, and GPT-5.2. The harness is open source, built on Pydantic AI and DuckDB, with automated grading against gold SQL and a published results dashboard, so the mechanics are inspectable end to end.

The result: querying through the semantic layer, GPT-5.3 Codex answered every trial correctly and Claude Sonnet 4.6 scored 98.2 percent, roughly 216 of 220 trials. The same models writing raw SQL over the same data scored 84.1 and 90.0 percent. Just as notable is what dbt disclosed alongside the headline. The text-to-SQL baseline was handed the entire schema in context, a favor the authors note does not scale to real warehouses. Several of the original questions sat beyond the semantic layer’s reach until a modest round of additional modeling brought them in, and that modeling improved both strategies. Disclosures like these are what make a benchmark worth taking seriously.

Before we put our own results on the page, one rule for reading the rest of this article: none of the numbers above can be ranked against any of the numbers below. The two studies use different tasks, different data, different denominators, and different definitions of correct. If a percentage from their study and a percentage from ours happen to look similar, the similarity means nothing by itself. What can be compared is the shape of the two findings.

The strongest finding in the dbt post is not a percentage. It is an asymmetry in how the two strategies fail. When text-to-SQL misses, it produces a plausible number that is quietly wrong: the query runs, the answer arrives formatted and confident, and nothing about it invites a second look. When the semantic layer misses, it is because the question fell outside its modeled scope, and the agent gets an error instead of an answer. One failure mode erodes trust invisibly; the other announces itself while there is still time to catch it.

We think dbt is exactly right about this, and it is the same argument we make from the provenance side. On Plexara, an answer arrives attached to the knowledge pages and catalog entries that shaped it, and the audit log records every call that produced it. The error message and the audit trail do the same job by different means: they turn failure from a silent event into an inspectable one. For a business deciding whether to let agents near real decisions, that property matters at least as much as the accuracy number, because it determines whether the residual errors are findable.

What dbt measures is coverage of a modeled scope. Their semantic layer is a set of predefined metrics over modeled warehouse tables, and the benchmark asks whether an agent that stays on that paved road beats an agent navigating the raw schema on its own. The questions are the kind a metrics layer exists to answer, and the win condition is matching the gold query over a shared, known dataset.

What we measure is resistance to traps. Our suite is built around questions that have a plausible wrong answer any competent agent will reach unless it knows a specific business fact: amounts stored in cents, revenue that means net rather than gross, a fiscal year that starts in February, a deprecated table that still looks authoritative. The knowledge that defeats each trap is taught to the platform and reviewed by a person, and the benchmark asks whether that taught layer prevents the confident wrong answer, and whether it keeps accumulating.

Complementary, not comparable. One study tests how well a curated road performs where the road exists; the other tests whether an agent can be kept from driving off a cliff the map does not mark. Both are real questions a buyer should ask. Neither number ranks the two products, because the two numbers are answers to different questions.

Our benchmark ladders through four platform configurations, from raw data tools with no context at all, through semantic enrichment, to the knowledge layer and search, to the full lifecycle, holding the model, the prompts, the seed data, and the 87-task suite constant across every arm and running each task three times. Because only the platform changes between arms, any difference in the answers is attributable to the platform. The method, figures, and reproduction commands live in the full benchmark report.

On knowledge-trap questions, the raw-tools baseline answered 42.7 percent correctly and the platform with its knowledge layer answered 98.7 percent, a gain of 56.0 points with a 95 percent bootstrap confidence interval of +44 to +67. The specificity matters as much as the size: on plain discovery and numeric questions, the four arms are statistically indistinguishable, so the platform is not making the agent generally smarter. It is supplying exactly the class of fact whose absence produces confident wrong answers, which mirrors the scope dbt drew around their own claim.

The instrument is built to be checked. Every number in the report carries a reproducible confidence interval, the resampling seeds are fixed, a threats-to-validity section says where the design is weakest, and a notebook regenerates every figure from committed data with no network access and no API key. The report and raw run data are archived on Zenodo, linked in the references below.

A second study asks the question the ablation cannot: where does the knowledge come from? Starting from a completely empty knowledge layer, where trap accuracy sits at 48.0 percent, a floor that reproduces across five independent runs, we taught the platform six business facts one at a time and measured after each. Accuracy climbed to 90.7 percent by the sixth fact, and the curve moved in legible steps: each taught fact lifted its own trap class at or shortly after the moment a reviewer promoted it to shared knowledge.

dbt found the same dynamic approaching from the other side. Their added round of modeling, three small dbt models, closed the semantic layer’s coverage gaps and improved both strategies. The convergent lesson is that a context layer rewards investment, and the two studies simply invest in different currencies: metrics modeled ahead of time in one, facts taught and human-reviewed along the way in the other. Either way, the layer an organization builds is an asset it owns and keeps improving, independent of which model happens to sit on top of it.

Set the two studies side by side, at the level of shape rather than score, and the agreement is striking. Both find that bare text-to-SQL access plateaus well short of reliability, at a level that varies with how hard the task leans on business context. Both find that a governed context layer moves the questions it covers into dependable territory. Both find that the layer changes the failure mode from a plausible wrong answer to something a person can inspect. And both find that the gain is scoped rather than general: dbt reports questions that fell outside the modeled layer, and we report suites where the platform makes no measurable difference.

That last point of agreement is the one we would most like buyers to notice, because restraint is rare in a contested marketing space. A benchmark built to sell would show a uniform uplift everywhere. Both of these show edges: places where the layer runs out, ties where context is not the bottleneck, disclosed favors to the baseline. Two teams independently choosing to publish their limits is a better reason to trust the category’s central claim than any single headline number either team produced.

The two products draw the layer around different territory, and the honest geometry is orthogonal with overlap. dbt’s semantic layer governs modeled warehouse data through predefined metrics, and the modeling discipline underneath it, transformation and metric definition, is work Plexara does not do. A warehouse that dbt has modeled is exactly the kind of source Plexara is built to govern: the two layers meet rather than compete.

Plexara carries the same thesis across a wider surface. The context layer spans federated SQL across warehouses and databases, object storage, REST APIs, and proxied MCP servers, under one persona model, one authorization decision, and one audit pipeline, with a knowledge layer that grows by being taught and human-reviewed in the flow of real work. The thesis both benchmarks support does not stop at the warehouse boundary, because the questions agents are asked do not stop there either.

Both benchmarks are public. dbt’s post and open harness are linked in the references, with a published dashboard of results. Our full report carries every figure with its confidence interval and the commands to reproduce them, and the archived report and run data live on Zenodo. Read both before you take either vendor’s word for it, ours included.