The alarm is an obituary
A monitoring alert is, by definition, an obituary. By the time it fires, the thing it was watching has already stopped — the system filled up, lost the thread, or crashed and never came back. You find out *after* the work has stalled, not before.
For most of computing history, that was an acceptable trade. Something breaks, an alert fires, someone fixes it. But autonomous AI changes the arithmetic. When you run a fleet of AI agents working around the clock — drafting code, processing documents, triaging tickets, moving data between systems — "we will find out when it breaks" quietly becomes "nobody noticed for three hours." There is no human sitting beside each agent. The alert is the only nervous system, and it only speaks once the patient is already gone.
We run exactly that kind of fleet of autonomous agents, and we kept hitting the same wall. Our monitoring was honest and accurate — and far too late. A watchdog would tell us an agent had gone silent, but by then it had been idle for hours, burning the calendar without producing anything. So we built something that answers a different question: not *what has broken*, but *what is about to*.
Why this is a business problem, not a dashboard problem
It is tempting to file this under engineering housekeeping. It is not. The cost of a stalled AI agent is rarely the agent itself — it is everything waiting downstream.
An agent that quietly stops mid-task does not throw an error you can bill to a vendor. It produces a half-finished report, a deployment that never completed, a customer query that sits unanswered until someone happens to look. The damage is the silent gap between the moment work stopped and the moment a human noticed. Reactive monitoring guarantees that gap exists by design — it cannot fire until the failure is already real.
For any organisation putting AI agents into production, this is the question that actually matters: not "how capable is the model?" but "when it drifts at 2am, how long until anyone knows, and is there still time to act?" Capability gets the headlines. Operability is what determines whether you can trust the thing unattended.
From post-mortem to forecast
The shift we made was from post-mortem to pre-emptive. Instead of waiting for an agent to cross a failure threshold, we score every agent continuously on how likely it is to stall *soon* — a single number, worst-first, so anyone scanning the board sees trouble building while there is still a live session to nudge.
The crucial word is *soon*. A reactive watchdog reports a fact that is already settled. A forecast reads the early signs of the same failure and surfaces it while intervention is still cheap — a quick check-in instead of a cold restart.
The four signs that an agent is drifting toward silence
We did not need new instrumentation to do this. The warning signs were already in the data we collected; the work was in reading them earlier and combining them. Four signals, in plain terms, tell you an agent is heading for trouble:
- Running out of working memory. Every AI agent has a finite context window — its short-term memory for the task. An agent that is nearly full, and filling fast, is far closer to losing the thread than one parked at the same level and idle. Rate of fill matters as much as the level.
- Declining quality of work. Not just the latest output, but the *trend*. Output that has been getting steadily worse over the last few turns is often the first visible symptom of an agent about to derail — the equivalent of a colleague whose answers are getting vaguer before they go quiet.
- Going quiet. Time since the agent last did anything, measured against what is normal for that task. At eighty per cent of the way to "this is too long," the risk is already high — this is the leading edge of the very signal the old watchdog used, just read earlier.
- A recent history of trouble. An agent that has already failed twice today is fragile. Recent crashes are one of the strongest predictors of the next one.
Each signal is weighted and blended into one score. The maths is deliberately boring. The intelligence is not in a clever algorithm — it is in choosing the right signals and being honest about them.
The discipline that makes a forecast trustworthy
The most important design decision was the least glamorous: what to do when we *do not know* something.
A freshly started agent has no quality history yet. An agent that is legitimately idle has no meaningful "time since last action." The naive move is to treat the unknown as "no risk" and lean harder on the signals you do have. That quietly inflates confidence — it makes an agent you know almost nothing about look reassuringly safe.
We did the opposite. When a signal is missing, we leave it out and do not redistribute its weight. The effect is that an agent we know little about *cannot* score as alarming as one we have full visibility into. We would rather under-warn on a cold start than cry wolf on every one.
That restraint is the whole game. A forecaster that fires constantly gets muted, and a muted forecaster is worse than none at all. Conservative beats clever for alerting — an alert you can safely ignore is an alert everyone will eventually ignore, including the one time it mattered.
A score is useless without a reason and an action
A number on its own does not tell anyone what to do. So next to each score we surface the reasons that actually moved it, in human language: *memory at 88 per cent, filling in about twelve minutes*, *quality dropping*, *silent for four of its five-minute budget*. A glance tells you whether to step in or wait.
And next to the reason sits the action — a single click that sends the agent a gentle check-in: *"Please save your progress and confirm your next step."* Often that one nudge is enough to pull it back on track before it goes quiet. It does not restart anything; it intervenes while the session is still alive.
The pattern generalises beyond AI: a score gets attention, a reason earns trust, and an action makes it useful. Drop any one of the three and people stop looking at the dashboard.
What changed, and what it means for you
The outcome was a shift in *when* a human gets involved — from discovering a dead agent and cold-restarting it, to watching a warning climb and heading it off. The old watchdog still exists as the last line of defence. The forecast simply moves the moment of intervention earlier, into the window where a nudge still lands.
Three lessons hold up well beyond our own stack, and they are worth carrying into any AI initiative:
- Predict from signals you already collect. We added no new plumbing. The leverage was in interpreting existing data earlier, not in building more of it.
- Conservative alerting is a feature, not a weakness. Refusing to over-state confidence on incomplete data is what makes the warnings trustworthy enough to act on.
- Capability is the easy half; operability is the half that earns trust. A model that can do the work is table stakes. Knowing, in real time, when it is about to stop — and being able to step in before it does — is what makes autonomous AI safe to run in a business.
If you are evaluating AI agents for real work, push past the demo. Ask how the system tells you it is in trouble, how early, and what you can do about it from where you are standing. The answer to that question, far more than the model's raw cleverness, decides whether you can leave it running.