Plexara
What you will take away from this lesson
211 - Governing the catalog without leaving the portal listed the lineage panel among the things a dataset's page shows. This lesson opens it. Lineage answers two questions people ask about every number: where did it come from, and what changes if the thing it came from changes.
On Plexara nobody draws it. Lineage is recorded by the platform as data moves through it: a query that writes a table, an automation that exports a file, a table registered over that file. Each edge says what wrote it and when.
Learning Objectives
- 01Name the three things that record lineage on Plexara: a SQL statement that writes a table, a managed script run that exports a file, and a table registered over a file.
- 02Follow a real chain on the ACME demo from five warehouse tables, through an automation’s export, to a table registered over it.
- 03Read lineage from the agent with fetch and in the portal on a dataset’s page, and say what each edge records about what wrote it.
- 04Explain how a column with no description of its own borrows one from the same-named column upstream, and how that borrowed curation is marked.
- 05State what lineage covers and what it does not: whole datasets and files, three hops each way, never individual columns.
Where we are in the curriculum
If any term in this lesson feels unfamiliar, the 100 series is one click back. The 200 series assumes that mental model.
100 Series: the foundation
- 101What is a Large Language Model?Brilliant at language, blind about your data. Tokens, hallucination, the grounding problem.
- 102Tokens and your budgetSubscription-plan economics, session limits, and Plexara enrichment dedup.
- 103Context, compression, and memoryThe keep / compress / clear playbook and how memory carries across sessions.
- 104Frontier models, specialized models, and why enterprise AI uses bothThree knowledge sources (training, web search, tools). MCP as the exposure protocol.
- 105What is an AI agent?The think/call-tool/observe loop. Professor's knowledge, child's literalism.
- 110Is MCP just an API wrapper?MCP as an application layer. Spectrum from thin wrapper to full application server.
If a term in this lesson looks unfamiliar, back up to the 100 series. The 200 series assumes that mental model. Every row above is a direct link.
Three things record lineage
Lineage drawn by hand is out of date the week after it is drawn. Lineage recorded by the system that moved the data is as current as the last time the data moved. Plexara records an edge each time one of three things happens on the platform.
What records an edge
| When the platform runs | It records an edge from | Writer |
|---|---|---|
| A SQL statement through trino_execute that writes a table from what it reads: CREATE TABLE AS, INSERT INTO SELECT, MERGE INTO, CREATE VIEW AS | Each table the statement read, to the table it wrote | sql, with the audited call that ran it |
| A managed script run that succeeds | Every table its queries read, to every file it exported (a portal asset or an uploaded file) | script, with the script that ran |
| A table registered over a file | The file, to the table | registration, with the registration id |
The second and third rows meet. A table registered over a script's export is reached, walking upstream, from the tables the script read, so a chain that crosses an automation and a spreadsheet-style registration is still one chain.
A chain on the ACME demo
The ACME demo runs a weekly automation, acme-supplier-margin, that compares what ACME pays for each product with what its supplier, Blue Harbor Wholesale, has quoted for next month. It reads four warehouse tables and the supplier’s price sheet, which was uploaded as a spreadsheet and registered as a table, and it exports a CSV of margin by category.
A merchandising analyst wanted to join that CSV to other warehouse data, so the agent registered it as a table too. Three steps, two kinds of writer, one chain.
One chain on the ACME demo
- Read
Five datasets on the acme connection
warehouse.public.transactions, transaction_items, products, categories, and the registered supplier price sheet.
script: acme-supplier-margin, weekly - Exported
supplier-margin-by-category
A CSV asset the automation writes on every run: margin now against margin at the supplier’s quoted cost, by category.
registration: reg_ab3e58474ef92be07f4eb85c7a60ec11 - Registered
scratch.uploads.admin_supplier_margin_by_category
A table over the CSV on the scratch connection. It follows the file, so each weekly run moves it onto the new numbers.
Nobody drew any of this. The script edges were recorded the last time the automation ran; the registration edge was recorded when the table was registered, before the call returned.
Reading it from the agent
Lineage is part of a dataset’s catalog record, so the agent reads it with the same fetch that returns the dataset’s columns and curation. There is no separate lineage tool to learn.
Reading the chain from the registered table
What the agent calls. This is the exchange the agent has with Plexara on your behalf, shown for the technical reader. You ask in plain language; you never type any of it.
fetch {
"reference": "mcp:dataset:(trino,scratch,scratch.uploads.admin_supplier_margin_by_category)"
}
→ "type": "registered_table",
"upstream": [
{ "source": "mcp:asset:871da875-cd99-44f9-b6d9-55732b4e9086",
"target": "mcp:dataset:(trino,scratch,scratch.uploads.admin_supplier_margin_by_category)",
"writer_kind": "registration",
"writer_ref": "reg_ab3e58474ef92be07f4eb85c7a60ec11",
"depth": 1 },
{ "source": "mcp:dataset:(trino,acme,warehouse.public.transactions)",
"target": "mcp:asset:871da875-cd99-44f9-b6d9-55732b4e9086",
"writer_kind": "script",
"writer_ref": "mcp:script:f3dc47f5-81e6-4fcd-a344-40dd494f86e2",
"depth": 2 },
{ "source": "mcp:dataset:(trino,acme,warehouse.public.categories)", ... "depth": 2 },
{ "source": "mcp:dataset:(trino,acme,warehouse.public.products)", ... "depth": 2 },
{ "source": "mcp:dataset:(trino,acme,warehouse.public.transaction_items)", ... "depth": 2 },
{ "source": "mcp:dataset:(trino,acme,scratch.uploads.admin_supplier_price_sheet)", ... "depth": 2 }
],
"downstream": [],
"query_availability": { "available": true,
"query_table": "scratch.uploads.admin_supplier_margin_by_category" }Trimmed from the ACME demo; each edge also carries who wrote it and when. The upstream list is the whole chain to the depth asked for, with depth counting hops from the dataset fetched. Every source and target is a reference the agent can fetch next, so walking the chain one step further is one more call.
Reading it in the portal
Anyone whose persona opens the catalog sees the same lineage on the dataset’s page, without asking the agent.
Every edge says what wrote it
An edge that only said "these two are connected" would answer where a number came from and nothing else. Each edge also says which automation, registration, or statement made the connection, as whom, and when, which turns "this table looks wrong" into a short list of things to check.
What every edge records
source, target
The two ends, as references: a dataset, an asset, or an uploaded file.
writer_kind
What recorded it: sql, script, or registration.
writer_ref
The specific writer: the script, the registration, or the audited SQL call.
written_by
The person the write ran as. For a script, its owner.
written_at
When the edge was last recorded.
depth
How many hops from the dataset being read, up to three.
The portal's lineage panel draws the same edges on the dataset's page, upstream and downstream, three hops each way, with each dataset a link that opens it in place.
Curation that follows lineage
A table registered over an export starts with no descriptions. Its columns often carry the same names as the columns they were computed from, and those upstream columns were described long ago. Plexara uses lineage to close that gap without copying anything.
How a column borrows curation along lineage
- 01
Only what is missing
A column keeps whatever description and glossary terms it has, and borrows only what it lacks.
- 02
Same name, nearest upstream
The lender is the same-named column, ignoring case, on the nearest upstream dataset that has curation, up to two hops away.
- 03
Never tags
A tag such as pii says something about the column itself, so tags are never borrowed.
- 04
Never written down
Borrowed curation is shown, not copied. Improving the upstream column changes what every downstream column shows.
- 05
Within your reach
An upstream dataset on a connection your persona does not reach lends nothing.
What the agent sees
On the ACME demo, the department column of the categories table carries a description and the Department glossary term. The registered margin table has a department column of its own and no curation on it.
A borrowed description, marked as borrowed
What the agent calls. This is the exchange the agent has with Plexara on your behalf, shown for the technical reader. You ask in plain language; you never type any of it.
trino_query {
"connection": "scratch",
"sql": "SELECT department, category, margin_now_pct, margin_projected_pct
FROM scratch.uploads.admin_supplier_margin_by_category
ORDER BY CAST(cost_change_pct AS double) DESC LIMIT 5"
}
→ "column_context": {
"department": {
"description": "The merchandising department a category rolls up to. One of 14
fixed values (Grocery, Household, Health, ...). Use it for rollups
when the 50 categories are too fine. ...",
"glossary_terms": [{ "name": "Department", ... }],
"inherited_from": {
"source_dataset": "mcp:dataset:(trino,acme,warehouse.public.categories)",
"source_column": "department",
"hops": 2,
"match_method": "name_exact" }
},
"margin_projected_pct": {
"description": "Gross margin percent if the quoted supplier cost replaced the cost
of record, holding sell price constant." }
},
"inheritance_sources": ["mcp:dataset:(trino,acme,warehouse.public.categories)"],
"rows": [
{ "department": "Garden", "category": "Flowers & Plants",
"margin_now_pct": "30.800", "margin_projected_pct": "20.600" }, ...Trimmed from the ACME demo. Nobody described department on the registered table. The query result carries the description and glossary term from the categories table two hops upstream, with inherited_from naming exactly where it came from, while margin_projected_pct shows the description written on the table itself.
What lineage covers
Lineage is most useful when you know exactly what it claims. Four statements cover it.
What lineage covers
Whole datasets and files
An edge joins a dataset or a file to another. Lineage never says which column fed which column; column curation is matched by name, as the rules above describe.
What the platform moved
Edges come from statements, runs, and registrations on Plexara. A pipeline that runs outside the platform records nothing here.
Three hops each way
fetch and the portal panel walk up to three hops upstream and downstream from the dataset being read.
What you can reach
A hop is shown only when the reader’s persona reaches both of its ends. A chain that crosses a connection you were not given stops at its edge.
Lineage in search
Where this leads
Key terms
Five terms cover the vocabulary of lineage on Plexara.
Key Terms
- Lineage edge
- A recorded link from one dataset or file to another that was made from it, with the writer, the person, and the time.
- Writer
- What recorded an edge: a SQL statement (sql), a managed script run (script), or a table registration (registration).
- Hop
- One edge. fetch and the portal walk three hops upstream and downstream; inherited curation reaches two hops upstream.
- Inherited curation
- A description or glossary term a column shows because the same-named column upstream has one and it does not. Marked with inherited_from, never written to the downstream column.
- Registered table
- A table over a stored CSV, JSON-lines, or Parquet file, cataloged when it is registered. Registration records lineage from the file to the table.