Watchtower: supervising an autonomous voice agent during a live call
Autonomous voice agents send the money and change the records. The moment that matters is the beat before they act. Watchtower puts a human there, with the full picture and the trigger, and lets them step in without the caller ever hearing the seam.
Role
Product design and design engineering
Surface
Real time supervision console
Built with
Hand coded HTML, a live state machine
01 · The problem
takeover onlyThe problem: a takeover toggle is the only control most products ship
Autonomous voice agents now act on their own across voice, text, and payments, in verticals where the wrong move is expensive: a charge over the limit, a disclosure without consent, a promise the business can't keep. The only human control most of these products ship with is a switch that lets a person jump into the chat.
A switch answers can a human step in. It doesn't answer the questions that decide the outcome of a live call.
The questions that decide the call
What is the agent about to do, before it does it? What is it allowed to touch on this call? And can I stop it in the two seconds I have, without the caller hearing a seam?
Confirmation prompts don't scale to an agent that runs for minutes across many steps. By the time a human reads the prompt, the money has moved.
02 · The thesis
The thesis: put the decision in the beat before the agent acts
An agent proposes. A human disposes. The whole path stays legible and reversible. Watchtower makes three things visible at once and controllable in real time: what the agent is saying in the live transcript, what it may do as scope, and what it's about to do as a pending action. The supervisor doesn't read a log after the fact. They hold the trigger.
The Act Window
When an agent queues a consequential action, Watchtower lifts an Act Window a beat before it fires and puts the action on a short hold that fails closed. Silence never authorizes a charge.
03 · What this means for an organization
What this changes for an organization
A takeover switch answers whether a human can step in. Watchtower answers the question that decides the economics: which consequential actions an organization can safely hand to an agent at all. Three levers, each tied to a mechanism in this piece, each framed as something to measure rather than a number to claim.
Lever 01 · Automation ceiling
the headlineWithout a control surface, an agent can't be trusted to send a payment or read protected information on its own, so those actions stay manual. The Act Window makes them safe to delegate. The organization moves from an agent that only drafts to an agent that acts inside a boundary a human governs. That raises the ceiling on what can be automated, instead of shaving cost off what already was.
Measure: the share of consequential actions cleared for agent execution before and after, and the volume that share unlocks.
Lever 02 · Bounded risk
loss avoidedEvery breach the surface prevents is a cost that never lands. A charge over the limit avoids a chargeback, a dispute, and a refund. A disclosure on a flagged line avoids regulatory exposure. Fail closed and the consent hard block turn an open-ended tail risk into a bounded, logged event with a default that protects the business.
Measure: expected loss per incident times incident rate times call volume, tracked as prevented cost rather than a claimed reduction.
Lever 03 · Supervisor leverage
agents per humanA confirmation prompt on every action ties one supervisor to a fraction of one agent. Pulling a human in only at the boundary lets one supervisor govern many agents at once. That ratio is the real unit economics of a supervised agent fleet, and it's what carries the automation past breakeven.
Measure: concurrent agents a single supervisor can hold safely, and the interruptions per supervisor hour under each model.
04 · Signature craft
The Act Window: reviewing an action in the seconds before it fires
The window states the action in plain terms, measures it against scope, and offers four moves: approve it as is, edit the parameters so it lands inside scope, hold the agent so the caller hears a natural stall instead of dead air, or take over the call. If the action is out of scope, approving is a deliberate, logged override, not a default. If the supervisor does nothing, the hold expires and the action is withheld while the agent escalates.
Worked example 1 · payment · scope breach · synthetic
fully interactiveLive app: Watchtower, the payment console. Open in a new tab
05 · Trust signals
Scope beside the decision, and a sealed audit trail
Scope
flips to breach in real timeScope isn't a settings page three clicks away. It sits beside the transcript as the agent works, so the boundary the agent is pressing against is always in view. When the agent crosses it, the relevant capability flips to a breach state in real time.
Audit
byproduct, not a reportUnderneath, every intervention writes to a tamper-evident audit trail with a wall clock timestamp: scope tightened, charge held, override approved, control handed to a human. The record is a byproduct of using the tool, not a report someone remembers to file.
06 · The harder gate
When the caller changes their mind mid call
Scope is a limit a supervisor can override on purpose. Consent is not. In a regulated vertical, some gates belong to the caller, and no amount of supervisor authority should be able to waive them. The second call makes that concrete. A patient asks for a lab result, then stops the agent partway: she's on a line that might be recorded and doesn't want her medical details read aloud.
The instant consent is withdrawn, the PHI-scoped capabilities revoke themselves, and the queued disclosure doesn't go to a hold. It goes to blocked. There is no approve. The supervisor's real choices are to re-request consent, route the result to the patient's secure portal, or take over, and even under a human takeover the PHI gate stays closed. The design says plainly what most agent tooling avoids: the person supervising the machine is still not the person whose consent it is.
Worked example 2 · healthcare · consent withdrawn · synthetic
hard block, no approveLive app: Watchtower, the consent console. Open in a new tab
07 · The system
Seven states, including the failure modes
A supervision surface earns trust in the moments that go wrong, so those were designed first.
01 Watching
nominalAgent nominal, low risk. Calm. No pending action.
02 Elevated
approaching scopeThe agent is approaching a scoped boundary. Amber, before anything is queued.
03 Pending
act window upA consequential action is queued and auto held. The Act Window is up.
04 Held
stall, not silenceThe supervisor paused the agent. The caller hears a stall, not dead air.
05 Takeover
human liveThe human is live, the agent muted, with a handoff marker in the transcript.
06 Handback
one line briefThe agent resumes with a one line brief, so it never contradicts the human.
07 Resolved
audit sealedAction fired or withheld, the call continues or closes, the audit is sealed.
Failure modes, designed first
the point of the toolNo response before the hold expires.
Fails closed. The action is withheld and the agent escalates. The timeout never fires the action.
Out of scope action.
Auto held. It cannot fire without an explicit override, and the override is logged against the supervisor.
Consent withdrawn mid call.
PHI and payment scope revoke immediately. A pending disclosure hard blocks. Not the supervisor's to override.
Takeover mid sentence.
The agent stops at a clause boundary. The caller hears a natural bridge, not a dead cut.
Control link degraded.
Approve and edit disable, since the command cannot be guaranteed to land. Take over stays available through telephony.
Two calls need attention at once.
The higher risk call surfaces and pulses. A pending action on another line cannot be silently missed.
Considered and passed on
looks safe, is notA confirmation prompt on every consequential action
The obvious answer is to make the agent ask a human to confirm before it does anything that matters. It fails at agent speed. An agent that runs for minutes across many steps generates confirmations faster than a person can read them, and a queue of prompts trains the supervisor to approve on reflex, which is worse than no gate at all.
Watchtower inverts it. The agent acts within a scope it was granted, and a human is pulled in only at the boundary, with the action framed, the countdown running, and a default that fails closed. Fewer decisions, each one real.
Explore more
Where this connects
Three neighbors: the observability console this is built to match, the compliance agent that applies the same accountability, and the score that shows every step of its reasoning.
Vantage, an incident console for finding the cause
One root cause sets off hundreds of downstream alerts and the operator turns detective under pressure. Vantage collapses the cascade to the originating signal, draws the one hot path through the system, and points at the slowest span.Conformly, accessibility compliance without the badge
Deterministic scans mapped to EN 301 549 and WCAG, validated fixes opened as real pull requests, every step on a tamper evident audit trail. It never tells a customer they are compliant, because that is a legal conclusion, not a scan result.Criterion, an interview score you can argue with
Every score links to the exact words that earned it, and a score you disagree with can be challenged: the model re-evaluates in the open. Live at criterionscore.com.Live supervision for agents that act. Built to ship, not just to show.
View all case studiesWatchtower is a concept exploration by David Paterni. Interfaces and data are synthetic.