Design the model-serving layer for a company with 15 LLM features. One gateway or per-team integrations, and what lives in it?
What they are really testing: Platform thinking. At one or two features, direct API calls are fine; at 15, the cross-cutting concerns, keys, budgets, fallbacks, observability, model migrations, either live in one place or are re-solved badly 15 times.
A real interview question
Design the model-serving layer for a company with 15 LLM features. One gateway or per-team integrations, and what lives in it?
What most people say
drag me
“Each team can call the provider SDK directly, it keeps them autonomous, and we can standardise later if it becomes a problem.”
At 15 features "later" already happened: keys in 15 places, no answer to what did we spend by feature, and the next model migration is 15 uncoordinated projects. Autonomy on prompts is healthy; autonomy on infrastructure is sprawl.
The follow-ups they ask next
How does a model migration actually roll out through this?
Per-feature: run the feature eval suite against the new model, canary a traffic slice, compare quality and cost dashboards, then flip the alias fully, with instant config rollback. No code deploys involved.
A team complains the gateway blocks a provider feature they need. What is the escape hatch?
Pass-through support for provider-specific parameters, fast gateway release cadence, and in the limit a documented direct-access exception with their own budget accountability. A platform that cannot say yes quickly breeds shadow integrations.
What the interviewer is listening for
- Justifies the gateway from concrete cross-cutting pain, not architecture fashion
- Model as routed configuration, migrations via canary and alias flip
- Keeps prompts with teams and treats the gateway as SLO-carrying infrastructure
What sinks the answer
- Either extreme unargued: full sprawl or a gateway that owns prompts and reviews
- No per-feature budgets or attribution in the design
- Ignores that the gateway is a new single point of failure
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“One gateway, because [keys, budgets, fallbacks, tracing and migrations] are otherwise solved [15 times or zero times]: [one internal API, model as config aliases], containing [credentials, per-feature budgets, health-aware routing, caching, uniform tracing], while [prompts stay with teams] and the gateway itself is [stateless, scaled, SLO-carrying].”
Keep going with architecture & design
Senior
Support reports the AI assistant got noticeably worse this week. Nothing was deployed. Walk me through the investigation.
Senior
When do multi-agent architectures actually earn their complexity over one well-tooled agent, and what fails in them?
Senior
Legal asks: what happens to our customer data when we use third-party LLM APIs, and what controls do we need? Answer them.
Senior
The CTO asks whether you should build your AI capability on frontier APIs or self-host open-weight models. How do you frame the decision?
Senior
Tell me about a time your eval data said one thing and an important stakeholder insisted the opposite. How did you resolve it?
Senior
Design a document Q&A system over all internal knowledge, wikis, drives, tickets, for a 10,000-person company.
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free