A named playbook that tells machine workers what to do after a failed step, partial state, or conflicting downstream effect.
The worker attempted a change but cannot prove whether the target system accepted, rejected, duplicated, or delayed it.
// AGENT OPS · MACHINE-WORKER MANAGEMENT · June 21, 2026
Agent Ops Need Recovery Playbooks Before Machine Workers Retry Broken Workflows
A retry is not a recovery strategy. It is an amplifier unless the worker knows exactly how to exit the failure.
Autonomous workers create operational risk when they treat failure as a prompt to keep moving instead of a signal to contain state.
Operators can let agents handle more work because failure paths are observable, reversible, and owned before production pressure hits.
Agent ops fails when recovery is improvised at machine speed.
The easiest agent demo shows the happy path: take a task, call tools, update systems, report done. Production does not fail on the happy path. Production fails when the third call succeeds, the fourth call times out, the fifth call reads stale state, and the worker decides to be helpful.
That helpfulness is the danger. A human operator pauses because ambiguity feels expensive. A machine worker often continues because its instruction says complete the job.
Agent ops needs recovery playbooks because autonomous workers will eventually meet broken integrations, duplicate events, partial writes, bad approvals, expired credentials, and inconsistent source systems. Without a recovery path, every failed workflow becomes a new workflow designed by the agent in real time.
Four failure states every machine worker must recognize.
Two systems now disagree after the agent moved one record, enriched one field, or consumed one event before another system caught up.
The next attempt could duplicate a charge, send a second message, overwrite a human edit, or reopen a closed workflow.
The accountable human, team, or approval queue is unavailable, but the workflow still has customer, revenue, or compliance consequence.
Example: the onboarding worker that recovered by creating more work.
A customer onboarding agent provisions a workspace, sends kickoff notes, assigns internal tasks, updates CRM stage, and opens a billing checklist. The workspace call times out after the backend has already created the account. The agent sees no success response, retries, and creates a second workspace.
Now the CRM points at one workspace, billing references another, customer email links to a third generated invite, and support gets two welcome events. Nothing looks catastrophic from any single tool response. Together, the company has created a customer-facing mess.
A recovery playbook would stop the chain. Unknown write means pause mutation, query by external id, compare receipts, quarantine duplicate objects, notify the onboarding owner, and only then resume from a named checkpoint. The agent does not invent recovery. It follows recovery.
The recovery playbook
Checkpoint
Define the last known safe state for the workflow: inputs received, records touched, messages sent, approvals used, and downstream jobs triggered.
Failure class
Classify the break as timeout, partial write, duplicate event, conflicting source, missing authority, unsafe retry, or downstream drift.
Allowed action
Name whether the worker may retry, pause, reconcile, quarantine, rollback, stage a fix, or escalate to an accountable owner.
Idempotency proof
Require external ids, receipt hashes, old and new values, dedupe keys, timestamps, and target-system confirmation before a retry is allowed.
Resume gate
Make restart conditional on a clean checkpoint, visible receipt, owner acknowledgement, and a record of what the recovery changed.
The point is not fewer failures. It is smaller failures.
Teams often overreact to agent failures by reducing autonomy everywhere. That is understandable, but it is a blunt instrument. The better move is to shrink failure domains and make recovery boring.
Good recovery playbooks let agents keep moving on safe classes of failure: read retry, stateless fetch, duplicate-safe enrichment, staged proposals. They force containment on dangerous classes: payments, customer messages, pricing, permissions, legal language, forecast changes, and destructive writes.
The operator advantage is simple: the company can scale machine work without pretending the machine will never break. It will break. The question is whether it breaks inside a playbook or inside the customer experience.
Autonomy without recovery is just incident response with a faster clock.
Operator checks
Three signs your recovery model is too weak
Workers retry based on missing success messages instead of proving what the target system actually did.
The team says agents can roll back, but nobody can show old values, downstream consumers, owner approvals, or side effects.
The failure path points to a Slack room, not a named owner, authority tier, SLA, and resume decision.
Install recovery before you expand autonomy.
Pick one agent workflow that touches customers, revenue, permissions, or production systems.
List every timeout, partial write, unsafe retry, duplicate event, split truth, and missing-owner condition.
Define exactly when the worker may retry, pause, quarantine, rollback, stage, or escalate.
Require receipts, idempotency keys, old and new values, affected systems, owner, and resume gate.
Concrete takeaway for this week
Take one production machine-worker workflow and write its recovery playbook before adding another tool permission.
- Name the checkpoint: the last safe state before each external write or customer-visible action.
- Name the failure classes: timeout, partial write, duplicate event, split truth, unsafe retry, and missing owner.
- Name the allowed move: retry, pause, reconcile, quarantine, rollback, stage a fix, or escalate.
- Name the proof: external id, receipt hash, timestamp, old value, new value, dedupe key, and downstream systems touched.
- Name the resume gate: who can restart the workflow and what evidence they must see first.
The blunt rule: if the agent cannot prove where the workflow broke, it does not get to decide how the workflow recovers.