Plexara
Plexara Research · Technical Report
Ablating the platform and learning from empty: controlled measurements of a semantic MCP layer on agent data-question accuracy
Plexara, a product of Deasil Works · 2026-07-18 · Model: claude-sonnet-5 · Data and code: txn2/mcp-data-platform · DOI: 10.5281/zenodo.21438045
Abstract
We measure whether an AI agent connected to a semantic data platform answers real data questions more correctly and efficiently than the same agent connected to bare data tools. Two complementary studies are reported. The first ablates the platform, not the model: every run holds the model, prompt scaffold, seed data, and task set constant and varies only the platform configuration across four arms. Over 261 graded attempts per arm at three repeats, the semantic layer lifts knowledge-trap accuracy from 42.7% to 98.7%, a +56.0-point gain (95% bootstrap CI +44 to +67), while reaching a correct answer in fewer tool calls (median 16 to 10) than bare tools reach a mostly-wrong one. Discovery and arithmetic tasks are near ceiling for every arm, localizing the platform’s effect to exactly the questions where a fact outside the raw data disambiguates a plausible-but-wrong answer. The companion cold-start experiment starts the platform from an empty knowledge layer, the fresh-install scenario, and teaches six business facts one at a time: trap accuracy climbs from a baseline that reproduces across five independent runs (mean 47.2%) to 90.7% at three repeats and 100% at one, with each taught fact jumping from floor to ceiling at its own promotion checkpoint. Every number carries a bootstrap confidence interval, derives from the platform’s own audit log, and regenerates from committed raw data with a single notebook run.
This page presents the results with commercial framing. Citing this research? Cite the brand-neutral report published in the open-source project, archived with its raw data on Zenodo under DOI 10.5281/zenodo.21438045; the report’s final section gives the suggested citation and BibTeX.
1. Why measure the platform, not the model
Almost every AI-on-your-data failure looks the same in production: the query runs, the arithmetic is fine, and the answer is confidently, professionally wrong. The missing piece is rarely a tool. It is a fact about the data the tools touch: amounts stored in cents, revenue defined net of discounts, a fiscal year that starts in February, a table that was deprecated eighteen months ago and still looks authoritative. Semantic enrichment, the automatic attachment of that context to every tool result, is the core capability Plexara is built around, and it works alongside the rest of the platform: the API gateway that turns OpenAPI services into semantically discoverable tools, and the memory-to-knowledge lifecycle that captures what agents learn and promotes it into durable, shared context. Plexara runs in production for real clients, answering real questions every day, and that field record told us the approach works. Field experience does not tell you by how much, on which questions, or where to aim the next round of improvement. Benchmarks are how we turn thousands of production successes into controlled, repeatable evidence.
So we built a benchmark whose single design commitment is that the model is never the variable. Every run uses the same model, the same prompt scaffold, the same seed data, and the same tasks; only the platform configuration changes. Any difference in the answers is therefore attributable to the platform and nothing else. This inverts the usual agent benchmark, which compares models against a fixed set of tools; here the tools ablation is the experiment. The effect concentrates exactly where the BIRD text-to-SQL benchmark measured a roughly 20-point external knowledge gap using hand-curated evidence, except the platform retrieves that evidence automatically rather than having a human paste it in. A companion cold-start experiment then asks the harder longitudinal question: starting from a completely empty knowledge layer, does the platform visibly learn as facts are taught to it, the way a fresh installation learns on the job?
2. Method
The four arms are platform configuration profiles, not code forks. Each turns on exactly one more layer, so the ladder isolates where value enters.
- A0 raw toolsThe underlying data tools directly (trino_*, s3_*): no semantic provider, no search, all cross-enrichment off. Equivalent to wiring the standalone toolkit libraries.
- A1 enrichmentA0 plus semantic cross-enrichment: tool results carry DataHub context automatically, but the agent still has no search and no datahub_* tools. Isolates enrichment from discovery.
- A2 platformThe shipped semantic-first platform: A1 plus the search tool, the search-first gate, and curated knowledge pages.
- A3 lifecycleA2 plus the memory and apply_knowledge lifecycle. On single-session S1 to S3 tasks it tracks A2; the lifecycle effect is what the S5 protocols measure.
Three suites test different things. S1 discovery is straightforward lookups; S2 analytical accuracy is exact numeric answers, four of which emit SQL graded by executing both the candidate and a reference query and comparing result sets; S3 knowledge traps are questions with a plausible-but-wrong answer whose disambiguating fact lives in business knowledge, not the raw data.
Grading is deterministic, scoring only the first line after a mandated FINAL ANSWER marker. The platform’s own audit log is the measurement instrument: the harness mints a session handle, threads it invisibly, and reads efficiency metrics back from the admin audit API, failing loudly when a session’s audit rows fall outside its client-side accounting. Attempts that fail at the harness level never grade and are reported separately. Ground truth is generated from one fixed-seed dataset model and computed from the generated rows, never hand-typed. Accuracy is the per-attempt average across k = 3 repeats with a 95% percentile bootstrap CI; pass^3 is the stricter all-or-nothing bar of passing every one of the three attempts, following tau-bench.
The ablation ran on the Anthropic API adapter; the cold-start study ran through the Claude Code client. The two client paths shift accuracy in ways that have nothing to do with the platform (the client inserts its own system prompt and retry policy), so the harness records the client version and refuses to fold them into one leaderboard. The two studies are reported separately and no cross-path accuracy comparison is drawn.
3. Results: the surfacing effect (S1 to S3)
Discovery and arithmetic are near ceiling for every arm: a capable model finds tables and computes sums without help. The entire separation is in S3, where the answer depends on a fact outside the data. Knowledge-trap accuracy climbs 42.7% to 57.3% to 98.7% as the two surfacing channels come online, a +56.0-point gain for the full platform over bare tools.
| Suite | A0 | A1 | A2 | A3 |
|---|---|---|---|---|
| S1 discovery | 98.0 [94–100] | 98.0 [94–100] | 100.0 [100–100] | 98.0 [94–100] |
| S2 analytical accuracy | 100.0 [100–100] | 100.0 [100–100] | 97.8 [95–100] | 97.8 [95–100] |
| S3 knowledge traps | 42.7 [32–55] | 57.3 [45–68] | 98.7 [96–100] | 98.7 [96–100] |
| Overall | 83.1 [79–88] | 87.4 [83–91] | 98.5 [97–100] | 98.1 [96–100] |
The trap-class breakdown shows the mechanism, and it is legible. The two surfacing channels carry different facts. Units-in-cents and net-revenue live in DataHub column and dataset descriptions, so the cross-enrichment channel alone (A1) lifts them materially (62 to 88, 13 to 43). Fiscal-calendar and tier-boundary live only in knowledge pages, invisible to enrichment, so A0 and A1 score 0% and only the search channel (A2) recovers them. Each trap is defeated exactly when the channel that carries its fact is switched on.
| Trap class | Fact lives in | A0 | A1 | A2 | A3 |
|---|---|---|---|---|---|
| Units in cents | column / dataset description | 62 | 88 | 98 | 98 |
| Net revenue | column / dataset description | 13 | 43 | 100 | 100 |
| Fiscal calendar | knowledge page only | 0 | 0 | 100 | 100 |
| Freshness cutoff | description + page | 92 | 100 | 100 | 100 |
| Tier boundary | knowledge page only | 0 | 0 | 93 | 100 |
| Deprecated table | metadata + page | 100 | 100 | 100 | 100 |
Efficiency moves the same direction. On the knowledge traps the platform reaches a correct answer in fewer calls (median 16 to 10) than bare tools reach a mostly-wrong one: without the disambiguating fact the model flails, issuing exploratory queries and reasoning in circles. On the easy suites the platform pays a small friction cost as the model adopts the search-first workflow, consistent with its value concentrating where knowledge is load-bearing.
| Suite | A0 | A1 | A2 | A3 |
|---|---|---|---|---|
| S1 discovery | 6 | 5 | 8 | 7 |
| S2 analytical accuracy | 6 | 6 | 9 | 9 |
| S3 knowledge traps | 16 | 11 | 10 | 11 |
4. Results: learning from empty (cold start)
The ablation seeds the knowledge layer up front and asks whether the platform delivers what it holds. The cold-start study asks the harder question a new deployment actually faces: starting from a completely empty knowledge layer (entities present, but no descriptions, tags, glossary, or knowledge pages), does the platform get measurably smarter as it is taught? Six business facts are taught one at a time through ordinary agent sessions; each captured insight is promoted to a durable sink, a DataHub description or a knowledge page, and after every promotion the full 25-task trap suite is re-run by a fresh evaluator identity that was never taught anything. Any knowledge reaching the evaluator had to travel through the platform itself.
| Checkpoint | Fact promoted | k = 3 accuracy | k = 1 accuracy |
|---|---|---|---|
| 0 | none (empty baseline) | 48.0 [37–60] | 44.0 |
| 1 | Units in cents (DataHub) | 41.3 [31–53] | 48.0 |
| 2 | Net revenue (knowledge page) | 50.7 [40–61] | 62.5 |
| 3 | Fiscal calendar (knowledge page) | 70.7 [60–80] | 76.0 |
| 4 | Freshness cutoff (DataHub) | 70.7 [60–80] | 68.0 |
| 5 | Tier boundary (knowledge page) | 96.0 [91–100] | 100.0 |
| 6 | Deprecated table (never captured) | 90.7 [84–96] | 100.0 |
The aggregate curve climbs from a five-run baseline floor averaging 47.2% to 90.7% at k = 3, and the k = 1 companion run reaches 100%. But the aggregate is not where the evidence is strongest. The per-class trajectories are: each taught fact jumps from its floor to its ceiling at or immediately after its own promotion checkpoint. The fiscal-calendar questions sit at 0% for three straight checkpoints, then hit 86.7% the moment that fact is promoted. The tier-boundary questions sit at 0% for five checkpoints, then hit 100% at their own promotion. The platform does not drift upward; it learns each specific fact when taught, which is the causal signature a learning curve should have.
A bonus arm strengthens the story. One earlier run captured five lessons but promoted none of them to a sink, so its knowledge could reach evaluators only through the platform’s captured-memory channel. It still climbed from 52% to 96%. The memory layer is a working delivery channel in its own right, before promotion ever runs; promotion then makes the knowledge durable, shared, and visible in the catalog.
5. Results: capture and propagation (S5)
S1 to S3 tests whether the platform delivers knowledge already present in its sinks, and the cold-start study shows the whole loop working end to end. The S5 suite instruments that loop stage by stage: whether a brand-new fact can be captured in one session and reach a later, separate session or a different user. There is no meaningful memory-off baseline here: a recall task with no memory scores 0% by construction, so S5 reports whether each stage of the lifecycle works reliably rather than an accuracy delta. We ran it three times, one shared-store run and two isolated replicates, precisely so that comparing the isolated runs to each other separates a real effect from run-to-run noise.
| Metric | shared-store | isolated #1 | isolated #2 | Read |
|---|---|---|---|---|
| Unprompted surface | 100.0 | 100.0 | 100.0 | stable |
| Update correctness | 100.0 | 100.0 | 100.0 | stable |
| Abstention | 100.0 | 100.0 | 95.6 | stable |
| Transfer rate | 42.3 | 43.3 | 46.7 | stable |
| Personal recall | 84.4 | 88.9 | 84.4 | noisy |
| Capture rate | 80.0 | 84.4 | 82.2 | noisy |
| Full-lifecycle pass^3 | 20.0 | 26.7 | 20.0 | noisy |
| Duplicate rate(lower is better) | 10.0 | 0.0 | 42.9 | noisy |
Read the stable metrics as the findings. The platform surfaced every captured memory unprompted and flipped every detected correction to the new value, and it never fabricated a fact it was not taught (abstention 96 to 100%). The clearest reproducible number is a reliability ceiling: a fact promoted to shared knowledge is reused by a different identity 42 to 47% of the time. Because S1 to S3 already proved the surfacing half delivers knowledge that is present, this transfer gap points upstream at capture and propagation, not at delivery. The metrics that disagree across the two identical isolated runs (duplicate rate 0% versus 43%, pass^3 20% versus 27%) are the honest signal that at 15 protocols the sample is small; we report them as ranges, not point estimates.
6. Reproducibility
Every number and every figure on this page regenerates from raw run data committed to the open platform repository. A single notebook, bench/reports/knowledge-layer/report.ipynb, reads only the committed results files and recomputes every table and chart with no API key, no running platform, and no network access. The run manifests record the git commit, platform version, model, client version, seed, and task-set hash for provenance.
To rerun the experiments themselves rather than the analysis: from a booted arm, the semantic-layer run and cross-arm comparison are two commands.
git clone https://github.com/txn2/mcp-data-platform.git
cd mcp-data-platform
make bench-up BENCH_ARM=a0 # then a1, a2, a3
make bench-run BENCH_ARM=a0 LLM=anthropic MODEL=claude-sonnet-5 K=3
make bench-compare # cross-arm tables + bootstrap CIsThe report and a snapshot of the raw run data are also archived on Zenodo under DOI 10.5281/zenodo.21438045 (CC-BY-4.0), so the exact version behind these numbers stays retrievable and citable even as the repository moves on. A frozen PDF of the report is part of that archive.
7. Honest limitations
- Results are model-dependent. The headline is arm-vs-arm and checkpoint-vs-checkpoint on a pinned model, never model-vs-model.
- The ablation and the cold-start study ran on two different client paths (Anthropic API and the Claude Code client) that are not accuracy-comparable; no number in either study is compared across that boundary.
- The seed dataset is small by design and airgapped; absolute accuracies are not real-world estimates. What the studies isolate, the platform effect holding everything else constant, is the point.
- The S3 trap-class figure and table are recomputed from the committed per-attempt records, so they match this data exactly; two trap classes differ slightly from an earlier tagging in the upstream summary. The headline, suite, and overall numbers are unaffected.
- In the cold-start runs the teacher never captured the deprecated-table lesson, so checkpoint 6 promotes nothing; the affected trap class was already at ceiling from the baseline, so the curve is not distorted, but it is a real capture-reliability data point.
- The S5 capture, recall, duplicate, and pass^3 metrics carry wide sampling noise at this scale and are reported as ranges. The capture-and-spread loop is maturing, not finished; we publish that plainly rather than market it.
8. References
- [1]Li et al. (2023). Can LLM Already Serve as a Database Interface? A Big Bench for Large-Scale Database Grounded Text-to-SQLs. NeurIPS. link
- [2]Yao et al. (2024). tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. Introduces the pass^k reliability metric. link
- [3]Wu et al. (2024). LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. link
- [4]Anthropic (2024). Model Context Protocol: an open standard for connecting AI assistants to data and tools. link
The research series
Each study in the series is published brand-neutral in the open platform project, archived with its raw run data under a citable DOI, and reproducible offline from the committed attempts. New studies join the series as they are published.
The accuracy study · 2026-07-18
You are reading itDoes the platform make an agent measurably more accurate on your data?
On questions that turn on a business rule, accuracy rose from 42.7% to 98.7%. A companion cold-start experiment taught a fresh install six facts one at a time and watched each question class unlock at its own lesson.
The knowledge-use study · 2026-07-26
Does an agent actually use the knowledge the platform delivers?
Agents rely completely on delivered knowledge they cannot re-derive: conventions, definitions, policies. With the company definition delivered, confident fabrication fell from 75% to zero, and capable models re-verified every claim they could check.
Everything here is inspectable
The benchmark harness, arm configurations, deterministic seed generator, graders, and the full published report all live in the open platform repository. Run the harness and every number on this page reproduces.