Describe a disagreement with a colleague about how complex an AI system needed to be. How was it resolved?

Question

Accepted Answer

A colleague designed our research-report feature as a 5-agent pipeline, planner, searcher, analyst, writer, critic, and I thought one agent with good tools and one review pass would match it at a fraction of the cost. His arguments were not silly: separation of concerns, per-stage prompts, and a pipeline diagram everyone understood. Rather than escalate opinions, we agreed on a bake-off with the terms fixed in advance, which is the part I would defend hardest: 30 real research tasks, blind grading by two colleagues against a rubric we co-wrote, plus cost and latency per task, one week to build each, and we wrote down before running it what result would settle it, because criteria agreed after results exist are just ammunition. The outcome was messier than either position, which is typical: quality was statistically indistinguishable on 26 of 30 tasks, his pipeline won clearly on the 4 hardest multi-source tasks, and mine was about 60 percent cheaper and twice as fast. So we shipped the single agent as the default path with his planner-plus-critic stages as an escalation tier for tasks the router classified as complex, roughly 15 percent of traffic. Neither of us had proposed that architecture; the data did. Two things made the disagreement productive rather than political: the timebox kept the stakes small, one week is cheap insurance against months on the wrong architecture; and blind grading meant neither of us could argue with the scores, only with the rubric, which we had co-written. The bake-off with pre-agreed criteria became the team standard for architecture disputes, and it has since killed two of my own proposals, which is the sign it is working.

Answer

What would you have done if there was no time for a bake-off?

Ship the simpler design instrumented to reveal where it fails, with the complex one as the named next step if specific metrics stall. Reversibility substitutes for certainty.

Answer

Why blind grading, colleagues can mostly guess whose output is whose?

Even imperfect blinding plus a co-written rubric moves the argument from taste to criteria, and disclosing the guess rate keeps everyone honest about how blind it really was.

Describe a disagreement with a colleague about how complex an AI system needed to be. How was it resolved?

Describe a disagreement with a colleague about how complex an AI system needed to be. How was it resolved?

The follow-ups they ask next

What the interviewer is listening for

What sinks the answer

If you genuinely do not know

Keep going with behavioural

Knowing the answer is not the same as recalling it under pressure