Logging Prompts and Responses Safely

Article summary

Log the input, output, model, parameters, token counts, cost, latency, and a request id, plus metadata like user and app version. Redact PII and secrets before the record is written, not after. Sample high-volume traffic and keep a fuller record of errors and low-quality outputs. Set an explicit retention window and delete on schedule. Logged traffic, once redacted, is the highest-signal source of evaluation data you have.

Logs are where observability meets a privacy problem. The prompt and response are the most useful things to record, because they let you reproduce any request and later turn real traffic into an evaluation set. They are also the most sensitive things you handle, because users put names, emails, medical details, and occasionally API keys into prompts. Doing this well means being deliberate about what you keep, what you strip, and how long you hold it.

What to log

For every model call, a good record carries:

Input: the fully assembled prompt, including system prompt and retrieved context, after redaction.
Output: the model's response, including any tool calls.
Parameters: model name and version, temperature, max tokens, and any other sampling settings.
Cost signals: input and output token counts, and the computed cost.
Timing: latency for this call, ideally per step if it is part of a chain.
Metadata: a request id that ties multi-step calls together, plus user id, session id, app version, and prompt-template version.

That last item, the versions, is what lets you answer "did the answers get worse after we changed the prompt on Tuesday?" Without a template version on each record, you are guessing.

Redact before you write

The rule that keeps you out of trouble: redaction happens before the record is written, never as a later cleanup pass. Once raw PII lands in an observability backend it has been replicated to indexes, backups, and dashboards, and you cannot reliably claw it back.

Run a detector over input and output and replace matches with typed placeholders:

raw:      "Email me at [email protected] about order 4471"
redacted: "Email me at [EMAIL] about order [ORDER_ID]"

Typed placeholders beat a blanket mask because they preserve the structure of the request, which is exactly what you need for debugging and for building eval sets. Scan for the obvious categories (email, phone, credit card, national id) and for secrets (API keys, tokens) that users paste in by accident. If you genuinely need the raw text for a support workflow, keep it in a separate store with tighter access and a shorter retention window than your general logs.

Sample versus log everything

Logging 100 percent of high-volume success traffic is expensive and low-signal, because most successful requests look alike. A better shape:

Sample successful requests at a rate you can afford, say 1 to 10 percent.
Log errors, timeouts, refusals, and anything that scored low on an online eval at close to 100 percent.

The failures are the records you will actually open, so bias your budget toward them.

Retention

Pick an explicit window and delete on schedule. Thirty to ninety days covers most debugging and eval needs; regulated data may demand shorter. Two rules make this defensible: the retention period is written down and enforced by an automated job, and it is consistent with what your privacy policy tells users. Do not keep logs "just in case" past the window you promised.

Logs become your eval set

The payoff for doing all this carefully is that redacted production logs are the highest-signal evaluation data you can get. They reflect what users actually ask, including the messy inputs you would never have invented. Pull a representative sample, keep the low-scoring and failed requests because they define your hard cases, label them against your rubric, and you have an eval set grounded in reality. The prompt eval methodology covers how to score it once you have collected it.

Frequently asked questions

Should I log every request or sample?

Both, at different rates. Log a sampled fraction of successful requests, since keeping every high-volume success is expensive and low-signal. But log errors, timeouts, refusals, and low-quality outputs at or near 100 percent, because those are the records you will actually open. Tie the sampling to the request id so a sampled request keeps its full trace and metrics rather than a partial one.

How do I handle PII in prompts I do not control?

Redact before the log record is written, not in a later cleanup job. Run a PII detector over the input and output and replace matches with typed placeholders like [EMAIL] or [PHONE]. Store a redacted copy for observability and, if you truly need the raw text, keep it in a separate, shorter-lived, access-controlled store. Assume anything in an observability backend will be seen by more people than the raw store.

Can I use production logs to build an eval set?

Yes, and it is the best source you have because it reflects real traffic rather than inputs you imagined. Pull a representative sample, redact it, and label the outputs against your rubric. The requests that failed or scored low are especially valuable because they define the edge cases your eval should cover. Just make sure the data is redacted and that your privacy policy and retention window permit this use.