LLM Agent Observability: The Five Traces You Must Log From Day One

When an agent gives a wrong answer at 2am, logs are the only thing standing between you and a guess.

Observability for ordinary services is a solved problem: response time, error rate, resource use. For agents those tools are close to useless, because the worst failures raise nothing. The agent responds successfully. The response is simply wrong.

So something else has to be recorded.

Why agent observability is not service observability

In a normal service you read the execution path from the code; in an agent, the execution path is itself an output, constructed at runtime.

That difference changes what a log is for. Service code is fixed, and the log only tells you which branch ran. An agent decides, per request, which tools to call, in what order, and when to stop. If those decisions are not recorded, reconstructing a failure is impossible — you have an input, a final output, and a black box between them.

Trace 1: the resolved prompt, after every substitution

Almost nobody sends the raw prompt written in the code. There is a template, variables get filled, retrieval results are appended, conversation history is stitched on.

What must be logged is the final string — exactly what reached the model. Most teams log the template instead, then spend hours hunting a bug that lived in one of the substitutions. A single irrelevant retrieved document can add four thousand tokens and change model behaviour without appearing anywhere.

Trace 2: tool calls with arguments and returned payload size

For every tool call, record three things: the tool name, the actual arguments, and the size of what came back.

Payload size is the one everyone forgets and the one with the highest diagnostic value. A query returning 500 rows instead of 5 raises no error and is not slow. It just fills the context window until the agent forgets the rest of its instructions. In the log, that is one number that quietly went up a hundredfold.

Trace 3: tokens and cost per step, not per request

Total request cost tells you the month was expensive. Per-step cost tells you which step was expensive.

In practice one step usually burns eighty percent of the spend, and it stays invisible until you measure at step level.

Trace 4: the escalation decision — and the one not taken

When the agent hands off to a human, log why: low confidence, missing authority, or an explicit rule.

Harder and more valuable: the cases that came close to the escalation threshold and did not escalate. If the confidence threshold is 0.7, keep every case that decided between 0.7 and 0.75 in its own bucket. That bucket is a preview of your next set of failures.

Why these five and not more

You could record twenty things, and most teams that take up observability do exactly that. The result is a volume of data nobody looks at.

These five were chosen because each answers a different class of question: what went in, what was done, what it cost, where it stopped, and whether it helped. Anything that does not serve those five questions gets dropped in practice.

A simple test for any trace you want to add: name the decision you would make differently because of it. No answer, no trace.

Trace 5: the user's next action

This is the only genuine quality signal, and it is usually not captured at all.

What did the person do after reading the answer? Close the window? Ask the same question in different words? Request a human? Actually take the recommended action? A rephrased question is the strongest available evidence that the answer was useless, and no automated evaluation will ever see it.

A worked example: the bug with no error

A support agent ran well for several weeks, then quality drifted down. No errors in the log. Latency flat. Monthly cost slightly up — enough to be written off as traffic growth.

Once cost was logged per step, the retrieval step turned out to have moved from about 900 tokens to 6,100. The cause was an innocuous change: someone had added the internal notes field to the index. Those notes were long and mostly irrelevant, they filled the context window, and the instruction "escalate if unsure" was effectively falling off the end of the prompt.

The escalation rate had dropped from 9 percent to 2. That single number explained the whole thing in three minutes — but only because it was being recorded.

The one id that stitches it together

Five separate traces are worth little on their own. What is worth something is reading one complete run from start to finish.

So generate a run id at the first moment of the request and carry it through every step, every tool call, and every downstream service log. It is the simplest item on this list, and if it is not there from day one, adding it later means touching every code path.

A practical test for whether you have it right: given a customer complaint and a rough timestamp, you should reach that run's full decision chain in under two minutes. If it takes longer, the observability is still incomplete.

What to strip before any of this is written down

The resolved prompt contains customer data. That makes your agent log a sensitive data store that nobody has written a policy for.

• Redact phone numbers, emails and national identifiers at write time, not at display time.
• For retrieved documents, log the id and the length, not the body.
• Set a retention window. Thirty to ninety days is enough for diagnosis.

The trade-off is real: the more you redact, the harder reconstruction becomes. The workable middle is keeping full text only for cases that ended in an escalation or a complaint.

Do I need a specialised tool for this?

Not at the start. One table with run id, step number, type, tokens, cost and payload size covers all five traces. Specialised tooling earns its place when the volume stops being reviewable by hand.

How expensive is logging full prompts?

Storage is cheap and negligible next to inference cost. The real cost is the liability of holding sensitive data, not disk.

Where do I start if the system is already live?

Traces 3 and 4. Per-step cost and escalation rate can both be added without touching the architecture, and they return the most signal for the least work.

Closing

An agent without these five traces is a system you can have opinions about. With them, it is a system you can repair.

This layer is most of what separates a prototype from an operational agent.