Design an AI code-review assistant for a 200-engineer organisation. What does it check, and how do you stop it becoming noise everyone ignores?
What they are really testing: A design prompt where the failure mode is social, not technical. Anyone can pipe diffs to a model; the design skill is precision management, comment budgets, confidence thresholds, and measuring dismissal rate, because a noisy reviewer gets muted org-wide in a month.
A real interview question
Design an AI code-review assistant for a 200-engineer organisation. What does it check, and how do you stop it becoming noise everyone ignores?
What most people say
drag me
“On every pull request, send the diff to a model with instructions to find bugs, security issues and style problems, and post the findings as review comments.”
This exact design has been deployed and muted a hundred times. No context beyond the diff, no precision control, no comment budget, and style commentary on top, it produces 15 comments per PR of which 2 matter, and engineers learn the ratio within a week.
The follow-ups they ask next
Why run the adversarial false-positive pass instead of just a confidence threshold?
Self-reported confidence is poorly calibrated. A second pass explicitly arguing against the finding surfaces the weak ones far more reliably, and it is cheap relative to the credibility cost of a bad comment.
An engineer disputes a finding and the model was wrong. What does the system do with that?
The dismissal plus reason feeds the rule eval set as a labelled false positive, weekly tuning reviews the accumulating cases per category, and repeated failures in one pattern retire that rule. Feedback must visibly change behaviour or people stop giving it.
What the interviewer is listening for
- Optimises precision over recall and says why muting is irreversible
- Context beyond the diff: call sites, conventions, intent
- Comment budget, dismissal-rate operations, and shadow-mode rollout
What sinks the answer
- Diff-to-model-to-comments with no precision mechanism
- Style commentary in scope alongside linters
- Success measured in findings posted rather than findings acted on
If you genuinely do not know
Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.
“Design against [noise, because a muted reviewer never recovers]: [three high-confidence lanes, style excluded], [context: surrounding code, call sites, conventions], [precision gate: adversarial false-positive pass, threshold, 5-comment budget], operated on [dismissal rate per rule with cuts at 30 percent], rolled out [shadow first, then pull-based].”
Keep going with system design
Senior
Design a document Q&A system over all internal knowledge, wikis, drives, tickets, for a 10,000-person company.
Senior
Design an AI customer support system for a SaaS product: 50,000 tickets a month, target 60 percent automated resolution without wrecking satisfaction.
Senior
Design the evaluation platform for a company running 12 LLM features. What does it provide, and how do teams use it without a central bottleneck?
Senior
Design an agent that automates invoice processing end to end: receive, extract, match to purchase orders, and schedule payment. Finance signs off on it. What does the design look like?
Senior
Design the model-serving layer for a company with 15 LLM features. One gateway or per-team integrations, and what lives in it?
Senior
Support reports the AI assistant got noticeably worse this week. Nothing was deployed. Walk me through the investigation.
Knowing the answer is not the same as recalling it under pressure
Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.
Start free