Retiring an integration platform one flow at a time

On Plexara the replacement for a flow is an automation: a script the agent writes, the platform stores and versions, and Deasil runs. The agent starts from what the flow does rather than how it was configured. It reads the source systems through the same governed connections a person would use, writes the transformation, and dry-runs it against real data with nothing kept. The runs it records become tests, so every later version is checked against them before it can be saved.

The new automation then runs beside the old flow on the same inputs, writing to its own table while the old flow keeps feeding production. That side-by-side period is the whole method. It turns "does the rebuild work" from a judgment into a comparison.

When the outputs agree, a person approves what the automation does from a plain-language summary of its behavior, without reading code, and the automation takes over the production write. The old flow is switched off first and removed later, so the rollback for the first few days is turning it back on.

A parity proof is a short written statement: what was compared, how much of it matched, and where every difference came from. It starts with row counts and key coverage, because a rebuild that writes the right values into the wrong number of rows fails in a way a spot check never sees. Then it compares every column of every matched row.

A proof that says "49,633 of 49,633 rows equal on every column" is easy to accept. The more useful proofs have a remainder. In one, 498 of 500 rows matched on all 30 columns, and the two that did not were traced to the source system itself, not to the rebuild. Explaining a difference is the actual work. A migration that stops at "close enough" carries an unexplained difference into production and turns it into somebody else’s incident.

The agent does most of this work. It writes the comparison queries, runs them, and drafts the explanation of each mismatch. A person reads the proof and decides whether the explanation holds. When the proof is written down, the cutover decision can be reviewed later by someone who was not in the room.

Rebuilding a flow from its behavior, then holding the rebuild to the old output, turns the migration into an audit. The comparisons surface behavior nobody had written down, and some of it was wrong.

One rebuild recovered pages of records the old flow had been dropping without an error, because its paginated read stopped early. A CRM sync that made five upstream calls for every change now makes one, which matters when the vendor meters its API. An invoice flow that sent records one at a time and lost failures silently now sends 200 per call and reports every record that fails. A report rebuilt as an automation stopped counting some sales twice, a fault the old report had carried for long enough that people had stopped questioning its totals.

None of those fixes were the goal of the migration. Each one came out of tracing a difference back to its cause instead of accepting it.

A public media organization has finished. Every flow on its integration platform, Apache NiFi, was replaced by an automation with a written parity proof, and the last process groups were disabled on October 22, 2026. The service no longer runs. Its sources span more than a dozen systems, including a CRM, a membership system, an email platform, analytics, video, streaming, events, and a television ratings vendor whose data is only available through a web portal with no API. The scheduled sync jobs it ran in its cluster were replaced in the same way and removed on October 15.

A national retailer is partway through, moving one flow at a time. Its CRM-to-ERP customer sync has run every two minutes since the old pollers were stopped on October 12. The invoice, warehouse-import, location-import, and card-settlement flows are replaced and tied out column by column with no differences. About thirty legacy reports have been rebuilt as automations. The remaining flows are in the queue.

Across both deployments that adds up to dozens of automations and thousands of runs a day, with well over 99 percent of scheduled runs succeeding. Staff at both organizations write and own some of those automations themselves.

Moving a flow is the first day of a long life. An automation that replaces a flow inherits everything the flow had to survive: vendor APIs that slow down, credentials that rotate, upstream systems that go offline at two in the morning. Plexara alerts an automation’s owner once when it starts failing and once when it recovers, never retries a failed run behind anyone’s back, and keeps vendor credentials out of the script entirely. Automations you can leave running covers that half.

Each automation also keeps its versions, its run history, and the approval that let each version change behavior. When someone asks in six months why a number moved, the answer is in the record and not in the memory of whoever did the migration.

NiFi is good software with a long record in production. Snowflake built its managed integration service, Openflow, on it. The reason to retire it here was not quality. It was that an integration platform is a second place where business logic lives, with its own operators, its own credentials, and its own idea of what the data means.

Moving each flow onto the platform where the data connections, the catalog, the knowledge, and the agent already are removes that second place. The CRM sync is now an automation the agent can read, explain, and change under review, beside the reports that depend on it. This is consolidation, one workload at a time, and the systems of record on either side of each flow stay exactly where they are.

Pick a flow with a clear output and a person who checks it: a nightly table load, a sync someone complains about, a report rebuilt every Monday. Ask the agent to read what it does and write the replacement. Run both for a week, prove parity, approve, and switch. The second flow goes faster than the first, because the connections, the conventions, and the comparison queries already exist.

The automations series walks through the mechanics: the agent writes the first script, how runs are started, and what a run may do, and the record it leaves. For how this compares to keeping a pipeline tool, see Plexara and pipeline tools.