What does observability look like for an LLM application, and how is it different from normal service monitoring?
What they are really testing: Operating maturity. Standard APM says the request succeeded; LLM observability must also say whether the output was any good, and the trace unit becomes the whole pipeline, retrieval, prompts, tool calls, not one HTTP span.
A real interview question
What does observability look like for an LLM application, and how is it different from normal service monitoring?
What most people say
drag me
“I would monitor latency, error rates and cost dashboards, and set alerts on API failures and spend spikes.”
Necessary and insufficient. Every serious LLM incident report includes "all dashboards were green": the request succeeded, the tokens were billed, and the answers were garbage. Without content traces and quality signals you learn about it from Twitter.
The follow-ups they ask next
Full traces contain user data and retrieved documents. How do you handle that?
PII redaction before storage, tight retention windows, access controls on the trace store, and sampling. The trace store is now a sensitive system, treat it like one.
What would page you at 3am versus wait for morning?
Pages: provider error spikes, latency SLO breach, cost runaway, guardrail trigger storm. Morning: gradual quality drift, feedback trends. Quality moves slower than availability, alert accordingly.
What the interviewer is listening for
- Traces the full pipeline per request, prompts versioned, chunks and tool calls included
- Monitors quality via sampled grading and implicit user signals, not just uptime
- Runs a daily canary eval to catch silent provider changes
What sinks the answer
- Believes green APM dashboards mean the feature works
- Cannot reconstruct what a specific past request saw and produced
- No mechanism to notice a silent model update
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Keep [classic latency, error and cost monitoring], add [full per-request traces: prompt version, chunks, tool calls, output], add [quality signals: sampled grading plus implicit user feedback as time series], and [a daily canary eval] because [providers change models under your feet with zero deploys].”
Keep going with observability
Mid
Users report your RAG assistant is giving wrong answers. Walk me through how you debug it.
Mid
What are hybrid search and reranking, and when does a RAG system actually need them?
Mid
What actually is an agent, mechanically, and what stops one from looping forever or going off the rails?
Mid
What problem does MCP, the Model Context Protocol, solve, and what should you check before plugging a third-party MCP server into your assistant?
Mid
Design the guardrails for a customer-facing AI assistant. What layers do you put around the model?
Mid
Your LLM bill doubled month over month with flat user growth. How do you find the cause and get it down?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free