How to Evaluate an AI Agent Before It Touches Production

Your agent passed the demo. That says nothing about the 200th ambiguous ticket.

An agent always looks good in a demo, because you chose the questions. The problem starts when it enters real traffic and stops being handed clean inputs. A customer writes half a sentence. A field on the previous record is empty. Two internal rules contradict each other.

An evaluation harness is the systematic answer to that.

What is an evaluation harness?

An evaluation harness is a fixed set of real inputs paired with their correct outcomes, run automatically against the agent before every release to produce one comparable score.

Three words carry the weight. Fixed, because a set that changes between versions produces numbers you cannot compare. Real, because the inputs come from operational logs, not from the team's imagination. Automatic, because anything that depends on someone remembering will eventually be skipped.

This is not a unit test. A unit test asserts one deterministic output. A harness measures a distribution: how many of fifty cases the agent handled, at what cost, and how often it stayed quiet.

Agents fail in four classes

After a few operational systems, the failures start rhyming. Almost everything lands in one of these:

The third class costs the most and leaves the least evidence. The system looks healthy because nothing threw. The only way to see it is to compare the output against an answer someone recorded in advance.

Build the golden set from real transcripts

Fifty real cases beat five thousand synthetic ones. Synthetic cases encode what the team imagines customers do. Customers do something messier, and the mess is exactly where agents break.

What works in practice: pull three months of operational logs and hand-pick fifty. Twenty routine, twenty genuine edge cases that actually occurred, and ten where the human on duty also got it wrong. Write the correct outcome for each, and record who signed off on it.

Budget about two days. It is the only part of this process whose value increases over time.

The staging ladder: offline, shadow, canary

No agent should reach operations in a single jump. Three rungs, each proving something different:

1. Offline evaluation. The harness runs against the golden set. It proves the logic is sound on known cases. It proves nothing about novel input.
2. Shadow mode. The agent runs on live traffic, but its output is logged rather than shown. One to two weeks is enough to see the real input distribution.
3. Canary. Five percent of live traffic with an instant rollback path. The first time an outsider sees the output.

Most teams skip the second rung because it feels like a delay. Shadow mode is precisely where the gap between demo and operations becomes visible, and it costs nothing in risk, because no user ever sees the result.

The scoring rubric and the stop rule

One number is not enough. Track four:

• Correctness: what share of the golden set produced an acceptable answer.
• Cost: average cost per case, not per month. A monthly figure blends traffic growth with efficiency decay and hides both.
• Latency: the 95th percentile, not the mean. The mean conceals the experience of your least patient users.
• Escalation rate: what share went to a human.

Escalation rate is the only metric with no good direction. Near zero means the agent is deciding things it should not touch. Very high means it is adding nothing. The healthy band has to be found per workflow.

Then the stop rule: write down what score blocks a release before you run it. Set the threshold after seeing the result and you will rationalise it every time. I carried this habit over from trading systems, where the same single rule separates risk control from the feeling of risk control.

A worked example: the invoice follow-up agent

A small finance team built an agent to chase overdue invoices. Read the record, check payment status, send a reminder. Flawless in the demo. Then they built a golden set from fifty real cases.

First offline run: 84 percent correct. Three of the eight failures shared a pattern — customers who had paid in instalments. The agent only saw the most recent transaction, never the sum, so it chased people who had already settled in full.

The demo never surfaced this, because nobody picks an instalment record for a demo. Shadow mode might have taken two weeks to reveal it. The harness found it in forty minutes, because the correct answer had already been written down.

The more useful detail came after the fix. Correctness rose to 96 percent, and the escalation rate went from 4 to 11 percent. The second number looks worse at first glance. It was healthier: the agent had learned to hand multi-payment records to a person.

The trade-off you are accepting

A harness is not free. The golden set carries a standing maintenance cost, and any real change of scope makes part of it obsolete. In practice, expect to rewrite roughly ten percent of the set every quarter.

The alternative is approving each release on a feeling. Feelings do not scale with traffic.

How often should the harness run?

On every prompt change, model change, and tool-definition change. If a full pass takes under ten minutes, run it on every merge. If longer, run it nightly.

Who owns the golden set?

Whoever owns operational output quality — not engineering. If the same team defines the correct answer and writes the code, the evaluation becomes a mirror.

Can another model score the evaluation automatically?

For soft attributes like tone, yes. For correctness, no. A judge model shares the failure modes of the model under test, and you end up with a reassuring number that measures nothing.

Closing

A demo shows what the agent can do. A harness shows how often it does not. The second number decides whether the system is dependable.

If you want to see how this layer gets built into a purpose-built agent, the custom AI agents page is the place to start.