How do you monitor AI agents in production?
Trace every run: record each model call, tool call and hand-off as a step in one trace, with its inputs, outputs, token use, latency and errors. OpenTelemetry's generative-AI conventions define standard spans for this, such as invoking an agent and executing a tool, though they are still marked as in development. On top of traces, track outcomes — tasks completed, escalations, user corrections — and review a sample of real runs regularly.
- Published
- 22 October 2026
- Sources checked
- 22 October 2026
- From
- Kramiva
01
Why logs are not enough
An agent's behaviour depends on what it saw at each step. A log line saying a run failed tells you little; a trace showing the model's input, the tool it chose, what the tool returned and what happened next tells you why. Anthropic's research team describes adding full production tracing as what let them diagnose why agents failed and fix problems systematically.
02
What to capture
- One trace per run, with a span per model call, tool call and hand-off
- Model, parameters, input and output tokens, latency, and errors per span
- Tool arguments and results, with sensitive fields redacted
- The final outcome and any human intervention
- Cost per run, derived from token counts
03
Standards
OpenTelemetry's GenAI semantic conventions define operations including create_agent, invoke_agent, execute_tool and invoke_workflow, and attributes such as gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Using them means traces can be sent to any compatible backend rather than one vendor's tool. Their status is still Development, so names may change.
04
Trade-offs
Full traces contain prompts and outputs, which may include personal or confidential data — they need the same access controls and retention limits as the data itself. Storing everything is expensive at volume; sampling saves money but can miss rare failures.
Where Kramiva fits
Production monitoring is part of Kramiva's AI automation service, and every decision is logged with what was proposed, on what basis and who approved it.
Sources
- Semantic conventions for GenAI agent and framework spans — OpenTelemetry
- How we built our multi-agent research system — Anthropic, 13 June 2025
- Demystifying evals for AI agents — Anthropic, 9 January 2026
Accessed 22 October 2026. Kramiva is not affiliated with these publishers.
Related questions
Tell us what you are building.
A short brief gets you a real reply from a founder within one business day — an honest read on fit, scope, and budget.