LLM Observability & Monitoring

A normal web request is easy to reason about. Same input, same output, one function, one log line if something breaks. An LLM request breaks all three assumptions. Feed the identical prompt twice and you can get two different answers. A single user question might trigger a retrieval, three model calls, two tool invocations, and a re-ranking step before anything comes back. And when the result is wrong, "wrong" is often a judgment call rather than a stack trace.

Observability is how you get a grip on that. It is the practice of instrumenting an LLM app so that after any request, good or bad, you can reconstruct exactly what happened and measure whether it was any good.

The three pillars still apply

Classic observability rests on three pillars, and they carry over.

• Traces capture the shape of a single request: every step, in order, with timing and parent-child relationships. For an LLM app this is the retrieval, the prompt assembly, the model call, the tool call, the response.
• Metrics are the aggregate numbers you watch over time: requests per minute, p95 latency, error rate.
• Logs are the detailed records of individual events, including the actual prompts and responses.

What changes is not the pillars, it is what you put in them.

The LLM-specific signals

On top of the classics, an LLM app produces signals that ordinary services never generate, and these are usually the ones that matter most:

• Token usage per call, split into input and output. This drives both cost and latency.
• Cost per request, derived from tokens and the model's price. A single user question can quietly cost fifty times more than a typical one.
• Latency per step, not just end to end. A slow response might be the retrieval, not the model, and you cannot tell without step-level timing.
• Tool calls: which tools fired, with what arguments, and what they returned.
• Retrieval details: which documents came back, their scores, and whether they were actually used.
• Eval scores: an offline or online quality judgment attached to the trace, so a low score points you at the exact request that produced it.

Capture these and a vague "the bot feels worse today" becomes a specific, inspectable trace you can open and read.

Observability is not evaluation, and not monitoring

These three words get used interchangeably, and keeping them separate makes the whole area easier to reason about.

• Observability is instrumentation. It answers "what happened on this request?" by producing traces, metrics, and logs.
• Evaluation is offline quality measurement against a fixed test set. It answers "is version B better than version A?" before you ship. Our prompt eval methodology and how to evaluate an LLM app cover this side.
• Monitoring is live alerting on the signals observability produces. It answers "is something wrong right now?" and pages you when it is.

The dependency runs one way. Evaluation and monitoring both consume what observability captures. If you never traced a request, you cannot evaluate it later or alert on it in production. So observability comes first.

Where to start

You do not need a platform to begin. Start by wrapping every model call so it emits a structured record: input, output, model, tokens, cost, latency, and a request id that ties multi-step calls together. That single change turns an opaque pipeline into something you can query. From there the rest of this cluster covers the specifics: tracing multi-step requests, logging prompts and responses safely, and monitoring cost, latency, and quality in production.