How do you choose which model to use for a new feature, and when do you pick the smaller, cheaper one?
What they are really testing: Whether model selection is a leaderboard lookup or an engineering decision. The tell is whether they evaluate on their own task and treat model choice as swappable, not an identity.
A real interview question
How do you choose which model to use for a new feature, and when do you pick the smaller, cheaper one?
What most people say
drag me
“I would check the benchmarks and pick the best model we can afford, probably the latest frontier model.”
Leaderboard reasoning. Public benchmarks measure their tasks, not yours, and defaulting to the biggest model bakes in maybe 10x unnecessary cost for tasks a small model handles.
The follow-ups they ask next
Why prototype on the expensive model if you plan to ship the cheap one?
It separates prompt problems from capability problems. If the frontier model cannot do the task, no prompt tuning on a small one will, and if it can, you have a quality target to right-size against.
What besides raw quality goes into the choice?
Latency and time-to-first-token, cost per request at projected volume, context length, tool-use reliability, data-handling terms, and rate limits.
What the interviewer is listening for
- Builds a task-specific eval before deciding
- Right-sizes: names task types where small models match big ones
- Keeps model choice swappable configuration behind a gateway
What sinks the answer
- Chooses by leaderboard or brand loyalty
- Defaults everything to the frontier model with no cost math
- Hard-codes one provider into the codebase
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Choose by [task requirements plus an own-data eval of 50 to 100 examples], not leaderboards. [Prototype on a frontier model to find the ceiling], then [test cheaper models on the same eval] and take the smallest that clears the bar, often [10x cheaper] on extraction and classification. Keep the choice [swappable config behind a gateway].”
Keep going with model selection
Foundation
What is a token, and why does it matter that models bill and limit by tokens rather than words?
Foundation
Why do language models hallucinate, and why can you not simply prompt them to stop?
Foundation
A model advertises a 200k context window. What can you actually rely on it for, and what not?
Foundation
What is an embedding, and what does "similar" actually mean when you search with one?
Foundation
Explain RAG to me, and tell me what problem it solves that a bigger model does not.
Foundation
What does temperature do, and how would you set it for a customer-facing extraction API versus a marketing copy generator?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free