Operators inspect the loudest, newest, or most politically sensitive output. The team feels busy, but it cannot say whether quality is improving or whether risk is merely hiding in the unreviewed tail.
// AGENT OPS · MACHINE-WORKER MANAGEMENT · June 6, 2026
Agent Ops Need Sampling Plans Before Review Becomes Guesswork
If review is just whoever checks whatever feels risky, the control plane is guessing. Sampling turns machine-worker oversight into a measurable operating system.
Review is not governance until it has a sampling plan.
Most agent teams start with blanket review. Every draft, route, enrichment, score change, and CRM update passes through a human because the company does not yet know what the machine worker can be trusted to do.
That phase is useful for learning. It is also impossible to scale. The moment agent volume rises, blanket review becomes a queue, then a rubber stamp, then a false sense of safety. Humans skim the obvious work, miss the subtle failure, and approve because the backlog is already late.
A sampling plan defines which machine decisions get inspected, how often, by whom, against which failure classes, and what happens to the worker's authority when the sample is bad. Without it, review is theater with a dashboard.
The control plane inspects representative work, high-risk work, and drift-prone work on purpose. Findings change permissions, policies, prompts, and routing rules.
The failure mode is quiet confidence.
A GTM research agent can be correct on easy accounts and wrong on accounts with ambiguous subsidiaries. A routing agent can handle normal territories and break when a strategic account changes ownership. A copy agent can write acceptable drafts while slowly reusing claims legal already rejected.
If humans only review the work that looks unusual, they never measure the ordinary stream. If they only review random work, they may miss the decisions with real blast radius. If they only review failures reported by downstream teams, they discover damage after it has already become cleanup.
Good sampling blends three lanes: random samples for baseline quality, risk-weighted samples for important decisions, and regression samples for known weak spots. That is how an operator learns whether a machine worker is earning more authority or just getting lucky.
What a useful sampling plan includes
Separate low-risk drafts, reversible updates, customer-facing sends, scoring changes, and source-of-truth writes. Do not review them as one blob.
Assign inspection frequency by risk and maturity: 5% for stable reversible work, 25% for disputed lanes, 100% for authority upgrades.
Track stale evidence, wrong owner, weak rationale, policy breach, duplicate work, buyer harm, and downstream cleanup separately.
Samples must change something. Clean samples can expand scope. Bad samples downgrade, pause, retrain, or force new receipts.
Grounded example
The outbound agent that looked accurate because only winners were checked.
A growth team runs an agent that researches target accounts, enriches contacts, drafts first-touch messages, and proposes sequence enrollment. Managers review a handful of strong-looking prospects each morning. The reviewed work is solid, so the agent earns more volume.
Two weeks later, sales complains about bad-fit accounts, duplicated contacts, and outreach to buyers already in active opportunities. The reviewed sample was biased. Humans checked the polished outputs that were easiest to like. They did not inspect rejected accounts, borderline accounts, old CRM ownership, or the long tail of normal enrollments.
The fix is a sampling plan: 10 random enrollments per day, every strategic-account touch, every account with stale CRM ownership, every source older than seven days, and a weekly regression sample against known duplicate-contact patterns. Now review produces governance instead of vibes.
The operating matrix
Lane
Bad sampling
Good sampling
Random
Only easy or visible outputs get checked.
A fixed baseline sample measures ordinary work and catches quiet drift.
Risk
Review waits for complaints or manager anxiety.
High-blast-radius actions are inspected before authority expands.
Regression
The same failure returns after each prompt tweak.
Known weak spots stay in the sample until the error rate proves stable.
Response
Findings become comments in a thread.
Findings alter permissions, rules, receipts, and escalation paths.
The point of review is not to approve more machine work. The point is to learn which work deserves trust.
Where sampling creates leverage
Cleaner source of truth
Sampling catches bad writes before CRM quality becomes a permanent cleanup project.
Less buyer damage
High-risk sends, duplicated contacts, and stale account context get inspected before agents scale the mistake.
Better authority calls
Operators stop arguing from anecdotes and start promoting or downgrading workers from measured failure patterns.
Concrete takeaway for this week
Build one sampling plan for one active agent lane.
- Name the decision classes: draft, propose, update, send, route, escalate, or write to source of truth.
- Set three sample lanes: random baseline, risk-weighted decisions, and regression checks for known failures.
- Define the failure taxonomy: stale evidence, wrong owner, weak rationale, policy breach, duplicate, or downstream cleanup.
- Tie samples to authority: clean periods earn scope; bad samples reduce permissions or force review mode.
- Publish the receipt: operators should see what was sampled, what failed, and what changed.
The blunt rule: if review findings do not change agent authority, you are not governing machine labor. You are just reading its output.