When Should an AI Agent Refuse to Act? Designing Escalation Boundaries

The most valuable thing your agent does all week is hand one case to a person.

In most projects escalation gets designed as the failure path: when the agent cannot cope, route to support. That framing turns escalation into a defect to be minimised.

Escalation is a capability, not a defect. Like any capability, it has to be designed.

Defining escalation

Escalation is the agent recognising that a decision sits outside its authority or its confidence, and handing the case to a named person with full context.

Each part of that closes a common failure. Recognising means escalation is not the by-product of a crash. A named person means the case does not land in an unowned queue. Full context means the human does not restart from zero.

Three triggers

Every escalation falls into one of three classes, and each runs on different logic:

Conflating the three is the most common design error. A large refund should not depend on model confidence. It should always escalate, even when the agent is certain. Confidence and permission are independent axes.

Define blast radius

The blast radius of an action is everything it would cost to undo if the action were wrong — money, time, and trust.

Three components, and usually only the first gets counted. Sending a wrongly worded apology email costs nothing and carries a large blast radius, because it cannot be recalled.

A workable way to rank tools is to ask three questions about each:

• Is it reversible? If not, the radius is high regardless of amount.
• Is it externally visible? Internal actions can be corrected quietly. External ones cannot.
• How many people does it touch at once? A bulk action multiplies its radius by the recipient count.

The escalation contract: what the human receives

This is where most implementations fail. The agent escalates, and a person gets a notification saying "needs review". Now the human has to reconstruct the entire context, which often takes longer than doing the task from scratch.

A useful escalation carries at minimum:

1. What the agent understood — the request restated in its own words.
2. What it checked — which records it read and what it found.
3. Why it stopped — which of the three triggers, with the number.
4. What it would have done, with a confidence level.
5. How the human's decision gets recorded, so it flows back into evaluation.

Item five is what turns escalation from a dead end into a learning loop.

Escalate to whom? The routing problem

"Escalate to a human" is an unfinished sentence. Which human? If the answer is "the support queue", you have converted escalation into delay.

Routing should follow from the trigger that caused it. Three triggers, three different destinations:

• Low confidence goes to subject knowledge — usually the specialist for that domain.
• High blast radius goes to approval authority — a role, never a named individual, or one person's holiday stalls the whole flow.
• Missing authority goes to whoever owns that policy, who is often outside the operations team entirely.

And one rule that gets forgotten: every escalation needs a deadline. A case sitting in a queue for four days is, from the customer's side, indistinguishable from a wrong answer. When the deadline passes, the system should escalate further on its own rather than waiting for someone to notice.

In practice, defining those two things — destination and deadline — is what separates escalation that works from escalation that merely moves responsibility around.

Read escalation rate as health, not as a failure count

Define escalation rate as a failure metric and the team will drive it down. The easy way to drive it down is loosening the threshold. You get a better number and a worse system.

The honest metric is escalation rate alongside escalation precision: of the cases escalated, what share genuinely needed a person? If the human agrees with the agent's suggested action eighty percent of the time, the thresholds are too cautious.

A worked example: the agent that never said no

An operations team launched an agent to answer customer questions about contract terms. For three months the escalation rate sat at 3 percent and everyone was pleased.

Then one case surfaced. A customer asked about a clause that did not exist in their contract. The agent found the equivalent clause in the standard template and explained it, without mentioning that this clause was absent from their agreement. The customer planned around it.

Reviewing the transcripts, the team found the agent had never once said "I don't know". Not because it always knew, but because there was no path for saying it — the only permitted output was an answer.

The change that fixed it was one rule: if the answer depends on a clause that was not found in this customer's own record, escalate. Escalation rate went to 11 percent. Complaints of that kind went to zero.

Two anti-patterns

The agent that escalates everything. Usually the result of cautious day-one thresholds that were never revisited. The team has built an extra layer between the customer and support.

The agent that never escalates. More dangerous, because it looks like success. An escalation rate near zero across a genuinely complex domain almost always means the path does not exist or the threshold is meaningless.

How do I set the confidence threshold?

Empirically, not theoretically. Sweep the threshold across your evaluation set and see how many real errors slip through at each point, and how many healthy cases get escalated needlessly. The right point depends on the relative cost of those two, not on a standard number.

Should escalation be visible to the user?

Yes. "I've passed this to a colleague, you'll hear back by tomorrow midday" builds trust. Silence and delay destroy it.

What if the human team has no capacity?

Then the problem is capacity, not agent design. Lowering the escalation threshold to hide a staffing gap moves the risk from your support queue onto your customer.

Closing

A good agent is not measured by how much it does. It is measured by the accuracy of the line it draws between what it does and what it declines.

That line gets designed. It is not discovered. This page shows where it lands in practice.