Design the evaluation platform for a company running 12 LLM features. What does it provide, and how do teams use it without a central bottleneck?
What they are really testing: A design prompt about the meta-system, and increasingly a real senior interview question. Tests separating shared harness from team-owned datasets, wiring evals into CI and production, and making the platform self-service.
A real interview question
Design the evaluation platform for a company running 12 LLM features. What does it provide, and how do teams use it without a central bottleneck?
What most people say
drag me
“A central evaluation service where teams submit their models and prompts for testing against curated benchmark datasets, with a quality dashboard for leadership.”
Three failures in one sentence: central curation makes the platform team the bottleneck and the owner of "what is good" for products they do not know, benchmark datasets measure nothing about these specific features, and a leadership dashboard is reporting, not engineering.
The follow-ups they ask next
Two teams eval the same underlying capability differently and disagree about a shared model upgrade. What happens?
Both are right for their features, upgrades are per-feature routing decisions through the gateway, not global flags. The platform makes the disagreement visible and shippable, not resolved by decree.
How does the platform keep LLM-judge costs sane across 12 teams?
Small judge models by default with calibration data justifying them, cached judgments for unchanged case-output pairs, sampling tiers for production grading, and per-team eval budgets visible on the same cost dashboards as everything else.
What the interviewer is listening for
- Platform owns machinery, teams own datasets and rubrics, stated as a principle
- One harness serving CI gates, release comparisons and production sampling
- Production-to-dataset loop and judge calibration as platform services
What sinks the answer
- Central team curates datasets and defines quality for products it does not run
- Evals exist only as a pre-release ceremony, no CI or production presence
- Benchmark datasets standing in for feature-specific measurement
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Split [machinery from meaning]: platform provides [harness, grader library, judge calibration, versioned datasets, baselines, dashboards], teams own [their cases, rubrics, thresholds]. One harness serves [CI gates, model comparisons, production sampling plus a daily canary], with [one-click trace-to-case] keeping datasets alive, all [self-service, measured by adoption].”
Keep going with system design
Senior
Design a document Q&A system over all internal knowledge, wikis, drives, tickets, for a 10,000-person company.
Senior
Design an AI customer support system for a SaaS product: 50,000 tickets a month, target 60 percent automated resolution without wrecking satisfaction.
Senior
Design an AI code-review assistant for a 200-engineer organisation. What does it check, and how do you stop it becoming noise everyone ignores?
Senior
Design an agent that automates invoice processing end to end: receive, extract, match to purchase orders, and schedule payment. Finance signs off on it. What does the design look like?
Senior
Design the model-serving layer for a company with 15 LLM features. One gateway or per-team integrations, and what lives in it?
Senior
Support reports the AI assistant got noticeably worse this week. Nothing was deployed. Walk me through the investigation.
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free