Skip to main content
Plexara
Field notes / product

Automations you can leave running

The first run of an automation is the easy part. The hard part is month six: a vendor API that slows down, a password that rotates, a job that fails at 2 a.m. and again every ten minutes after. Here is how Plexara keeps automations your agent wrote safe to leave alone.

8-minute readProduct

The six months after the first run

An agent can write a working automation in an afternoon. The report lands, the sync runs, the numbers tie out, and everyone moves on. Then the world the automation depends on starts to change. The vendor API gets slower on Monday mornings. The service account password rotates. A partner system goes down for maintenance at two in the morning, and a job scheduled every ten minutes fails at 2:00, 2:10, 2:20 and every ten minutes until someone wakes up.

Whether automation is worth doing is decided in those months, not on day one. A job that breaks quietly is worse than no job, because people keep trusting its output. A job that breaks loudly, forty times, trains everyone to ignore its alerts. The platform underneath has to handle the ordinary failures of a long-running system so the automation itself can stay simple.

One alert per failing streak

When a scheduled run fails, or a run started by an incoming event, Plexara opens an incident for that automation. The failed run that opens it sends one message to the automation’s owner and to the channels your admins chose for run failures: which automation, why it failed, the head of the error, and a link to the run. The failures that follow while the incident is open are counted and send nothing. A job that fails every ten minutes overnight sends one message, not forty-eight.

The next successful run closes the incident and sends one recovery message: when the failing began, how long it lasted, how many runs failed, and the error line and run that started it. A single failure followed by a success sends exactly two messages, which is the right number. The recovery message also says what it cannot know: a successful run means the automation works again, not that the work the failed runs skipped has been done. Whether that work needs redoing is the owner’s call, and the message is written so they make it.

A run someone starts by hand reports its failure to that person in the answer they are already reading, and alerts nobody else. Paging an on-call channel about a test someone is watching is noise.

One failing streak, two messages

  1. 01:40
  2. 01:50
  3. 02:00
  4. 02:10
  5. 02:20
  6. 02:30
  7. 02:40
  8. 02:50
  9. 03:00
  10. 03:10

Failure message

Which automation, why the run failed, the head of the error, and a link to the run. Further failures are not mailed.

quiet

Recovery message

The run that succeeded, when the failing began and how long it lasted, how many runs failed, and the error line and run that started it.

An automation scheduled every ten minutes. The first failed run opens an incident and sends one message to the owner and the run-failure channels. The five failures after it are counted and send nothing. The next successful run closes the incident with one recovery message. A run someone starts by hand reports its failure to them and alerts nobody else.

Why a failed run is never replayed blindly

The tempting fix for a failed run is to run it again automatically. Plexara deliberately does not, and the reason is the most important idea in this piece: a run that failed may already have written. It may have inserted half its rows, posted to a channel, or called a partner’s API before the step that broke. Running the whole thing again can duplicate exactly the work that succeeded.

Retrying is safe where nothing has happened yet, so that is where the platform retries. When a vendor refuses a call because of its rate, or a read meets a brief outage, the run waits the interval the vendor asks for and repeats that one call, up to three times, before handing the final answer to the script. A write that the vendor answers with an outage is not repeated, because the service may have acted on part of it before it answered. A whole failed run is never repeated on its own. The next scheduled run, or the next event, runs as normal.

Automations that need to pick up exactly where they left off keep a record of how far they got, saved only when a run succeeds. A failed run leaves that record untouched, so the next run reads from the same point and nothing is skipped or done twice.

Where retrying stops

Nothing happened yet

Retried inside the run

  • A call refused with “too many requests”, whatever its method
  • A read that met a brief outage
  • Waits the interval the vendor names, up to three attempts
  • The script sees the final answer and decides what to do with it

Something may have happened

Never repeated on its own

  • A write the vendor answered with an outage: it may have acted on part of it
  • A whole run that failed, scheduled or started by an event
  • The owner is told once, with the error and a link to the run
  • The next scheduled run or the next event runs as normal
A single call that the vendor refused for its rate, or a read that met a brief outage, can safely be asked again: nothing happened. A whole run that failed may already have written rows, sent messages, or called a partner, so running it again is a decision for someone who can see what it left behind.

What breaks later, and what absorbs it

Most long-term failures are not bugs in the automation. They come from the systems around it, and each has a platform answer that needs no change to the script.

Fragile vendors get a ceiling. An admin can cap how many calls run against a connection at once, across every automation and every agent session, so a busy morning cannot overwhelm a vendor that tolerates four concurrent requests. Each automation also chooses what happens when a run is still going as the next one comes due: overlap is allowed, the newer run is refused, or it waits in line and runs when the first finishes.

Credentials stay out of the code. Vendor passwords, API keys, and the seeds behind one-time login codes live in a write-only store in the admin portal. A connection or automation names a secret, and its value is filled in as each request leaves the platform. It never appears in the script, in a recorded call, in an audit record, or in an error message. When the password rotates, an admin changes it once and every automation using it picks up the new value on its next call. For a vendor portal that demands a one-time code at sign-in, the platform can produce the current code, so an automation does not stop at the login screen.

Exports refuse to be quietly short. An export cut off by a limit nobody asked for is refused with the reason, rather than delivering a file that looks complete. A cut the script asked for is written and marked Incomplete on the file, in its version history, and on the run that produced it.

Upgrades do not count as failures. Deasil upgrades every deployment as releases ship. Runs that are executing during an upgrade finish, or go back on the queue and start again, and an upgrade never records a run as failed.

Months two through twelve

A vendor API starts answering “too many requests”

The run waits the interval the vendor asks for and repeats the call, up to three times, before the script sees the answer

A fragile vendor falls over when several runs hit it at once

An admin caps how many calls are in flight against that connection at once, across every automation

A slow run is still going when the next one is due

Each automation chooses: runs may overlap, the later run is refused, or it waits in line

A vendor password rotates, or a portal asks for a one-time code

The credential lives in the admin’s secret store and is filled as each request leaves; rotating it touches no script

An export hits a row or size limit partway through

A cut nobody asked for is refused with the reason; a cut the script asked for is written and marked Incomplete everywhere it appears

Deasil upgrades the deployment while runs are executing

Running work finishes, or goes back on the queue to start again; an upgrade never records a run as failed

The failures that arrive after launch, and what the platform does about each without anyone rewriting the automation.

Long runs and the agent that started them

Some work takes longer than a conversation should wait: a large export from a vendor API, a run that walks a year of history. Large exports can run in the background while the conversation moves on, and the next session is told when they have finished. Agents that support the task extension of the Model Context Protocol get a handle for a long run or export the moment they start it, can check on it, and can cancel it. Either way the work belongs to the platform, not to the conversation that asked for it.

In production

Across two production deployments, dozens of automations run thousands of times a day, and well over 99 percent of scheduled runs succeed. When a run fails, the platform records the cause and whether a later attempt could succeed, which separates a vendor that was briefly down from a script that needs fixing. Most of the failures that remain trace to two outside sources: a vendor portal with no API that the automation signs in to like a person, and a CRM that times out under load. Both are exactly the cases the alerting is built for: one message when the streak begins, one when it ends.

What it does not do

This is the alerting and the resilience, not an incident-management product. There are no on-call rotations, escalation policies, or phone calls. Failure and recovery messages go to the owner by email and to the notification channels your admins choose, Slack, Mattermost, a webhook, or an email list, and a team that already runs a paging service can point a webhook channel at it.

Read more on Automations and Notifications, or start with what a run may do, and the record it leaves. For why the work belongs in a script at all, see the report that runs without the agent.