The board asks: we spent 2 million dollars on AI this year, what did we get? How do you make AI investment measurable, before and after the spend?
What they are really testing: Whether they can connect AI work to business value with intellectual honesty, counterfactuals, adoption versus impact, the difference between a demo and a deployed change, and impose that discipline before money is spent, not retrofitted after.
A real interview question
The board asks: we spent 2 million dollars on AI this year, what did we get? How do you make AI investment measurable, before and after the spend?
What most people say
drag me
“I would present metrics per initiative, tickets deflected, engineering hours saved, satisfaction scores, and translate them into dollar estimates to show overall return on the 2 million.”
It measures after the fact with no baselines or counterfactuals, so every number is contestable, "hours saved" is famously inflatable, and a board that senses inflation discounts the entire program, including the real wins.
The follow-ups they ask next
A popular initiative shows high usage but no measurable business delta after two quarters. What do you tell the board?
Exactly that, with the decision attached: renew with a sharper metric and a deadline, redesign the deployment, or wind it down. Popularity without impact is a cost centre with fans, and saying so is what makes the rest of the report believable.
How do you value defensive spend, guardrails, evals, monitoring, that prevents bad outcomes?
As risk reduction with reference points: incident rates before and after, cost of the incidents peers have had, compliance requirements met. Frame it like security spend, the board already knows how to reason about insurance.
What the interviewer is listening for
- Measurement designed into funding, metric, baseline and counterfactual named upfront
- Reports kills as portfolio health, not buried embarrassments
- Separates adoption from impact and labels estimates as estimates
What sinks the answer
- Retrofits dollar claims onto initiatives with no baseline
- Hours-saved arithmetic presented as hard return
- A portfolio where everything is claimed to have worked
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Measurement is [a funding condition, not a retrospective]: every initiative names [metric, baseline, target, counterfactual] upfront, deltas come from [holdouts and phased rollouts, not before-after], and the board sees [a portfolio: wins with defensible numbers, kills counted as health, platform valued by what it accelerates], with [adoption never sold as impact].”
Keep going with strategy
Senior
The CTO asks whether you should build your AI capability on frontier APIs or self-host open-weight models. How do you frame the decision?
Principal
Every quarter brings a new AI paradigm the company is urged to adopt. As the senior AI voice, how do you decide what the organisation adopts, watches, or ignores?
Principal
You have 8 product teams all building AI features independently. Do you build a central AI platform team, and if so, what does it own?
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free