What is a token, and why does it matter that models bill and limit by tokens rather than words?
What they are really testing: Whether you have actually called a model API and watched the meter, or only chatted with a UI. Tokens drive cost, context limits and latency, which makes them the unit an AI engineer budgets in.
A real interview question
What is a token, and why does it matter that models bill and limit by tokens rather than words?
What most people say
drag me
“Tokens are like words, the model reads text as tokens.”
It is the chat-user answer. It names the concept without any of the consequences, and every consequence, cost, context and latency, is the part an employer is paying you to manage.
The follow-ups they ask next
Why are output tokens priced higher than input tokens?
Input is processed in one parallel pass, output must be generated sequentially token by token, which holds the hardware longer per token.
Your prompt fits the window but quality drops when it is very full. Why?
Attention degrades over long contexts, models miss things in the middle. Fitting is not the same as being used well.
What the interviewer is listening for
- Quotes rough numbers, 4 chars per token, output costlier than input
- Connects tokens to all three: cost, context, latency
- Mentions logging token usage from the start
What sinks the answer
- Thinks a token is exactly a word
- Has never looked at a bill or usage dashboard
- No idea the context window is shared with the output
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“A token is [the chunk the tokenizer splits text into, about 4 characters]. It matters because [billing is per token], [the context window is a token budget shared with the answer], and [output is generated token by token, so length is latency].”
Keep going with llm fundamentals
Foundation
Why do language models hallucinate, and why can you not simply prompt them to stop?
Foundation
A model advertises a 200k context window. What can you actually rely on it for, and what not?
Foundation
What does temperature do, and how would you set it for a customer-facing extraction API versus a marketing copy generator?
Junior
The model has no memory between API calls. How do chat products remember the conversation, and what goes wrong as it grows?
Foundation
What is an embedding, and what does "similar" actually mean when you search with one?
Foundation
Explain RAG to me, and tell me what problem it solves that a bigger model does not.
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free