A client rang me last quarter, rattled. Their new "autonomous" support agent had spent a weekend issuing refunds to anyone who asked firmly enough. No malice, no breach — just a model doing exactly what it was told, with a tool that could move money and no one watching the door. The bill was five figures. The lesson was cheaper than most: an agent is not a feature you ship, it's a coworker you onboard.
The rush to put agents into production has outpaced the discipline to run them safely. Everyone wants autonomy. Almost no one wants to write the guardrails first. That order is backwards.
Autonomy is a dial, not a switch
The framing that gets teams in trouble is binary — either a human clicks every button, or the agent runs free. Real systems live on a spectrum, and the smart move is to start low and earn your way up.
I use four rough tiers with clients:
- Suggest: the agent drafts, a human sends. Zero blast radius.
- Approve: the agent proposes an action; a human clicks yes before anything executes.
- Act-and-log: the agent executes reversible actions on its own, every step recorded.
- Full autonomy: reserved for narrow, well-understood tasks with hard limits.
Most production agents should sit at "approve" or "act-and-log" for months, not days. Promote a workflow only after it has earned trust with real traffic. The refund agent should never have touched the top tier on day one — refunds are irreversible money movement, the exact case that demands a human in the loop.
Scope the blast radius before the capabilities
The dangerous question is "what can this agent do?" The right question is "what's the worst thing it can do, and how bad is that?" Capability and consequence are different axes, and you govern the second one.
Give an agent the narrowest tools that still let it finish the job. Every extra permission is a liability you now have to reason about at 2 a.m.
Concretely, before an agent goes live I want three things nailed down:
- Least-privilege tools. A read-only reporting agent gets read-only credentials. It cannot write, delete, or pay — not because we trust it, but because we removed the option.
- Hard limits in code, not prompts. "Please don't refund more than $200" in a system prompt is a suggestion. A check in your tool wrapper that rejects refunds over $200 is a rule. Never enforce a boundary in the one place the model is free to ignore it.
- Reversibility gates. Irreversible actions — deletions, payments, external emails, production deploys — cross a line that requires either human approval or a cooling-off window.
Observability is the whole job
You cannot govern what you cannot see. An agent that acts without a legible trail is not a productivity tool, it's a liability generator. In my engagements, the teams that sleep well have made every agent decision auditable after the fact.
That means logging the full trace: the input, the model's reasoning where available, which tools it called with which arguments, what came back, and the final action. When something goes sideways — and it will — you need to reconstruct exactly what happened without guessing. Tie every agent action to a trace ID and a human owner. "The AI did it" is not an incident report.
Watch for the quiet failures
Loud failures are easy; the agent crashes, you get paged. The expensive ones are quiet: the agent that confidently does the wrong thing, over and over, and no one notices until the numbers drift. Set behavioral alerts, not just uptime checks. Refund volume spiking 4x? Tool-call error rate climbing? A sudden change in the mix of actions? Those are your early warnings.
Governance is a practice, not a document
A policy PDF nobody reads is theater. Governance that works lives in the pipeline: a kill switch that any on-call engineer can hit, rate limits that cap damage per hour, staged rollout so a bad change hits 1% of traffic before 100%, and a routine review of what your agents actually did last week.
Regulators are catching up fast — the EU AI Act and sector rules across the GCC increasingly expect demonstrable oversight of automated decisions. The upside is that the controls that keep you out of trouble legally are the same ones that keep you out of trouble operationally. Build them once.
Where to start Monday
Pick your riskiest agent — or the one you're about to ship — and answer three questions in writing: What's the worst single action it can take? Is that action enforced in code or merely requested in a prompt? If it misbehaves for an hour, how fast can you see it and stop it? If any answer makes you wince, you have your first week of work.
Autonomy is something an agent earns, not something you grant on launch day. Govern first, and the autonomy takes care of itself.