You probably know
more than you can
say out loud.

What gets you hired

A token is the unit the tokenizer splits text into, roughly 4 characters of English, so 1,000 tokens is about 750 words. It matters because it is the unit of everything I have to budget. Cost: APIs price per token, and output tokens typically cost 3 to 5 times input tokens, so a verbose system prompt is cheap compared to letting the model ramble in its answers. Context: the window, say 200k tokens, is shared between the system prompt, conversation history, retrieved documents and the response, so if my RAG pipeline stuffs 50 chunks in, I am spending both money and the model attention budget. Latency: output is generated token by token, so response time scales with answer length, which is why capping output length and streaming are the first two latency levers I reach for. In practice I log token counts per request from day 1, because cost surprises in LLM systems are almost always token surprises.

What gets you hired

A model is a next-token predictor: it produces the most plausible continuation, and it has no built-in mechanism that checks plausibility against truth. When the true answer is well represented in training data, plausible and true coincide. When it is not, the model still produces something fluent, because fluent is what it optimises for. That is why prompting "do not hallucinate" barely moves the needle: you are adjusting style, not adding a fact source. The fixes that work all change what the model conditions on or how its output is checked. Ground it with retrieval so the answer is generated from supplied documents, and require citations so claims are traceable. Give it an explicit out, "say you do not know if the context does not contain it", which in my experience cuts fabricated answers dramatically because the model needs permission for the low-probability response. Constrain output to a schema where possible, an enum cannot invent a 13th category. And for high-stakes flows, add a verification step that checks claims against the source before the user sees them. I treat hallucination as a property to engineer around, like network failure, not a bug to prompt away, and I measure it with an eval set of maybe 100 known-answer questions rather than assuming.

What gets you hired

The 200k is capacity, not guaranteed attention. The model will accept that much input, but retrieval quality inside the window is not uniform: models are consistently better at using material near the beginning and end, and miss facts buried in the middle, the lost-in-the-middle effect. So I rely on it for headroom: long conversations, big retrieved bundles, whole files when needed. I do not rely on it as a substitute for retrieval. Stuffing 150k tokens of documents into every request costs maybe 50 to 100 times what a well-targeted 2k-token retrieval costs, adds seconds of latency, and still gives worse answers than sending the 5 passages that matter placed prominently. My working rules: retrieve and rerank rather than dump, put instructions and the highest-value context early, keep an eye on the middle for anything critical, and if a use case genuinely needs long-context recall, test it with a needle-in-haystack style eval on our own data rather than trusting the benchmark. Capacity went up 100x in three years; the discipline of sending only what matters did not change.

What gets you hired

At each step the model has a probability distribution over next tokens. Temperature rescales it before sampling: near 0, probability concentrates on the top token so outputs become highly consistent; around 1, the distribution stays broad so plausible alternatives get sampled and outputs vary. For the extraction API I would run at 0, or as close as the provider allows: the product promise is that the same invoice produces the same JSON every time, variance is a bug, and schema drift breaks downstream parsers. For the marketing generator I would run around 0.8, because there the product promise is variety, a user clicking regenerate wants a genuinely different headline, and at temperature 0 they would get near-identical output on every click. Two caveats I would flag. Temperature 0 is not a correctness setting: it confidently picks the most probable token, and if the model is wrong, it is deterministically wrong. And even at 0, providers do not guarantee bit-identical outputs across calls or model updates, so anything that needs true reproducibility needs it enforced outside the model, in tests and schemas, not in a sampling parameter.

What gets you hired

Each API call is stateless: the model remembers nothing between calls, so memory is something my application constructs by resending context. Naively that means the full transcript goes in every request, and that has two failure modes. Cost: input grows every turn, so turn 30 costs many times turn 1, and a long chat can be 10x the cost people budgeted. Quality and capacity: eventually the window overflows, and well before that, attention over a sprawling history degrades, the model starts missing constraints stated 40 turns ago. So real products curate. The standard toolkit: keep a sliding window of the recent turns verbatim, since recency matters most; maintain a running summary of older turns, compressing "what has been established" into a few hundred tokens; pin critical facts, user preferences, decisions made, so they survive summarisation; and for memory across sessions, store history externally and retrieve relevant pieces per query, which is just RAG over past conversation. Provider prefix caching softens the cost of the stable early history but does not fix attention decay, so curation still wins. My practical defaults: cap history hard, summarise beyond the cap, and test the feature at turn 50, not turn 3, because that is where memory features actually break.

What gets you hired

An embedding is a list of numbers, typically 1,000 to 3,000 dimensions, produced by an embedding model trained so that texts with similar meaning end up close together in that space. You search by embedding the query and finding the nearest stored vectors, usually by cosine similarity. The crucial word is meaning: "how do I reset my password" will match "steps to recover account access" despite sharing almost no words, which is the whole win over keyword search. But the same property cuts the other way: exact strings that carry no distributed meaning, error codes, part numbers, a person named "March", retrieve badly, because the embedding model has no idea that ERR-4512 must match exactly. That is why production retrieval is usually hybrid: dense embeddings for semantics plus BM25-style keyword matching for exact terms, with the two result lists fused. Two more practical notes: embeddings from different models live in incompatible spaces, so changing your embedding model means re-embedding the entire corpus, and similarity scores are relative, a 0.78 is not universally "good", so thresholds have to be tuned per corpus rather than copied from a tutorial.

What gets you hired

RAG is a two-step pattern: given a query, first retrieve the most relevant passages from your own corpus, usually with embedding search, then have the model generate the answer grounded in those passages. The reason it exists is three problems that no amount of model scale fixes. First, freshness: any model has a training cutoff, but a RAG index can be updated in seconds, so "what changed in yesterday policy" is answerable. Second, private data: your company wiki, tickets and contracts were never in training data, and RAG is how the model sees them without you shipping your data into a training run. Third, verifiability: because the answer is generated from identifiable passages, you can cite sources and let users check, which you cannot do with knowledge baked invisibly into weights. It also tends to be the cheapest option: keeping knowledge in an index costs an embedding call per document, versus fine-tuning runs every time anything changes. The trade is a new failure surface, retrieval quality. In most RAG systems I have debugged, the model was fine and the retriever was the problem, wrong chunks in the top 5, so the answer was wrong no matter how good the generation step was.

What gets you hired

Not necessarily, and I would decide on numbers, not the tool landscape. First question: how many vectors and what query rate. A docs site or internal wiki is typically 10k to a few 100k chunks. At that scale pgvector inside the Postgres we already run is my default: with an HNSW index it comfortably serves single-digit-millisecond similarity queries into the millions of vectors, and it keeps embeddings next to the source rows, so there is no sync pipeline that can drift, metadata filtering is a WHERE clause, and backups and permissions are the ones we already operate. A dedicated vector store earns its complexity when the numbers say so: tens of millions of vectors and up, high QPS with heavy filtered search, or a need for index tuning the extension cannot offer. The cost people forget to price is operational: a separate store is a second system to secure and monitor, plus an ingestion pipeline keeping it consistent with the source of truth, and that pipeline is a classic silent-failure spot, deletes that never propagate, so search keeps returning documents that were removed. So my decision rule is: pgvector until measured retrieval latency or scale forces a change, and if we ever migrate, it is behind a retrieval interface so the app does not care.

What gets you hired

Chunking is a precision-versus-context trade: too big and retrieval gets diluted, a 4,000-token chunk matching on one paragraph drags 3,800 irrelevant tokens into the prompt; too small and chunks lose the context that makes them answerable, a row of numbers with no idea what table it came from. My approach: split on structure first, headings, sections, paragraphs, because document authors already encoded meaning boundaries, and only fall back to fixed-size splitting, somewhere in the 300 to 800 token range, for unstructured text. Never split mid-sentence or mid-table. The naive failures are predictable: a fixed splitter severs the answer from the question it belongs to, strands "it" and "this policy" from whatever they referred to, and cuts tables in half so neither piece is usable. Three practices fix most of it: prepend document title and section heading to every chunk so it stays self-describing after retrieval, use 10 to 15 percent overlap so boundary-straddling facts survive, and for long sections consider embedding a summary that points at the full section. Then measure instead of arguing: a retrieval eval, 50 real questions with known source passages, checking whether the right chunk lands in the top 5. Chunking changes move that number more than model changes do.

What gets you hired

They fix two different failures. Dense retrieval has a blind spot for text whose importance is exact rather than semantic: error codes, SKUs, invoice numbers, people and product names, legal section references. A user searching "ERR-4512" or "clause 14.2" can get semantically-adjacent fluff while the exact match sits unranked. Hybrid search fixes that by running classic keyword scoring, BM25, in parallel with vector search and fusing the two ranked lists, usually reciprocal rank fusion, so exact matches surface even when embeddings shrug. If your corpus and queries are identifier-heavy, support tickets, contracts, codebases, hybrid is close to mandatory; for purely conversational corpora it moves less. Reranking attacks a different problem: first-stage retrieval optimises for speed over a whole corpus, so its ranking within the top candidates is mediocre. A cross-encoder reranker reads query and document together and scores relevance far more accurately, but at maybe 50 to 200ms for a few dozen candidates, so the pattern is retrieve 50 fast, rerank to a top 5 worth putting in the prompt. That improves answer quality two ways: better chunks first, and fewer tokens sent, which cuts cost. When do you need them? When measurement says so: if recall at 50 is fine but precision at 5 is poor, that is the reranker signature; if known-item queries fail, that is hybrid. I would not install either until a retrieval eval shows which failure we actually have, because each is another latency and operational cost.

What gets you hired

A system prompt is the standing configuration for the conversation: who the model is, what task it is doing, the constraints, the output format, and how to handle edge cases like off-topic requests. Stable things belong there, and it pays real dividends to keep it stable because providers cache the repeated prefix, cutting its cost dramatically, often around 10x on the cached portion. Per-request content, the user question, retrieved documents, belongs in messages, clearly separated. What it cannot do is guarantee anything. It is a strong prior, not an enforcement mechanism: a determined user can often talk the model out of its instructions, and text inside retrieved documents or tool results can carry injected instructions that compete with mine. So my rule is that the system prompt handles quality and behaviour, tone, format, refusal style, but every property with a security or correctness stake is enforced outside the model: output schemas validated in code, permissions checked by the application not the prompt, and sensitive-data rules applied by filters that run regardless of what the model decided. If a requirement appears only in the system prompt, I treat it as unenforced.

What gets you hired

Context engineering is treating the model input as a budget you assemble per request, and both the selection and the order are decisions. The components, in the order I place them. First the stable block: system instructions, tool definitions, and any fixed few-shot examples, stable-first is not stylistic, providers cache the repeated prefix and charge cached tokens at a fraction of the price, often around a 10x discount, so a byte-identical opening block is directly money and latency. Then the semi-stable: summarised long-term memory or user preferences. Then per-request material: retrieved knowledge, curated hard, and recent conversation turns. The current question and its immediate context go last, both because it is the natural reading order and because attention is strongest at the start and end of the window, which is also why anything truly critical, output format, safety constraints, lives in the system block, and why I never bury a must-follow rule in the middle of 20 retrieved chunks, the lost-in-the-middle zone. Selection is where quality is won: retrieval brings candidates, but what enters the prompt should be the reranked best few, say 5 chunks, not the top 30, because irrelevant context does not just cost tokens, it actively dilutes, the model grounds on plausible-but-wrong passages; history gets a sliding window plus a running summary rather than the full transcript; and few-shot examples earn their place, 3 well-chosen ones usually beat 10. The discipline that makes it engineering rather than folklore: ablate against the eval suite, drop a component, measure, and in my experience most mature prompts carry passengers, an instruction paragraph or an example block whose removal changes nothing except cost. The compact rule: every token either helps this request or it is noise wearing a seatbelt, and the eval, not intuition, says which.

What gets you hired

Treat prompts as what they are, production logic with an unusual editor population, and give them the code lifecycle with friction tuned so iteration survives. Versioning first: prompts live in source control as templates with typed variables, and every change is a diff with an author and a reason, the dashboard-edited prompt that nobody can reconstruct after an incident is the baseline failure here. Review and testing: every prompt edit triggers the owning feature eval suite automatically, results on the change request, a 100-case suite runs in about 5 minutes, and the gate is regression against baseline with per-slice deltas, because prompt edits classically fix the case someone was staring at and quietly break three others. For the 2 of my 5 editors who are not engineers, the interface matters: a playground where they iterate freely against sample cases, and a submit path that runs the same eval gate, review means the suite passed plus a colleague glance, which keeps PMs iterating without shipping vibes. Deployment: prompts deploy like config, decoupled from code, versioned, with canary, new version on say 10 percent of traffic with quality signals watched, and rollback as a config flip measured in seconds, which converts the worst prompt incident from an emergency deploy into a toggle. Tracing closes the loop: every production request logs its prompt version alongside model and parameters, so "answers got weird Tuesday" resolves to "v41 shipped Tuesday 14:00, here is the diff" in minutes, and A/B comparison of prompt versions on live traffic becomes possible because attribution exists. Two disciplines around the machinery: an owner per prompt, five people editing 30 prompts without ownership is how contradictory instructions accrete; and a monthly prune, because prompts grow by accretion of special cases until nobody knows which sentences still do anything, the eval suite is what makes deleting a sentence safe.

What gets you hired

I would push back, because this confuses the two axes. Fine-tuning changes behaviour, style, format, how the model responds. It is bad at implanting facts: knowledge from a small fine-tune is recalled unreliably, mixed with hallucination, with no way to cite where an answer came from, and the docs change, so you are signing up for a training run per release. What "know our docs" actually needs is retrieval: chunk and index the docs, retrieve the relevant passages per question, and generate answers grounded in them with citations. Freshness becomes an index update measured in seconds, and answers become verifiable. My rule of thumb: knowledge lives in the index, behaviour lives in the weights or the prompt. Fine-tuning earns its place when the problem really is behaviour, matching a house voice, sticking to a strict output format the base model fumbles, or distilling a well-understood task onto a small model to cut cost, say replacing a frontier-model call with a fine-tuned small model at maybe a tenth of the price once the task is stable. I would also say the boring part out loud: start with RAG plus a good prompt, measure with an eval set of 50 to 100 real questions, and only reach for fine-tuning when that measurably plateaus.

What gets you hired

My rule: fine-tuning is the right call when the task is stable, the cheaper ladder is exhausted, and the economics justify it. The ladder first, because iteration speed dominates: a prompt change ships in minutes, few-shot examples in the prompt teach most formats, and RAG handles anything that is really a knowledge problem. Fine-tuning iterates in days and produces an artifact you must version and maintain, so it has to earn that. The cases where it genuinely wins: consistent style and format the base model cannot hold, a house voice, strict domain-specific output structure; distillation for economics, once a task is stable and well-measured on a frontier model, tuning a small model on its outputs often gets you 95 percent of the quality at maybe a tenth of the cost and much lower latency, at high volume that is real money; token efficiency, when few-shot examples eat a large prompt every call, baking the behaviour in removes them; and narrow domain language where the base model reads it clumsily. Doing it properly is mostly a data and measurement exercise. Data: hundreds to low thousands of examples that look exactly like production inputs and ideal outputs, deduplicated, cleaned, with a held-out split, and every mislabeled example teaches the wrong thing, so quality beats volume. Measurement: the eval suite exists before training, runs on the base model for a baseline, then on the tuned one, and includes off-task checks, because tuning can degrade general behaviour outside the target task, the regression nobody looks for. Then production reality: the tuned model is a versioned artifact tied to its data snapshot, re-evaluated and re-tuned as the task drifts, and the base-model-plus-prompt path stays available as a fallback. If a team cannot name the metric fine-tuning should move, they are not ready to fine-tune.

What gets you hired

I treat it as an engineering decision with a measurement, not a leaderboard lookup. First I pin down the task requirements: what does a good output look like, what error rate is tolerable, what latency does the UX need, what is the cost budget per request. Then I build a small eval before choosing anything: 50 to 100 real examples with expected outputs, because public benchmarks tell me about benchmark tasks, and my extraction job or support bot is not one of them. I prototype on a frontier model first, deliberately, to learn the quality ceiling and get the prompt right. Then I run the same eval against smaller and cheaper models. The pattern I see repeatedly: classification, extraction, routing, summarisation of clean text, a small model matches the big one within a couple of points at maybe a tenth to a twentieth of the price and half the latency, so the small model wins. Multi-step reasoning, ambiguous instructions, and agent orchestration are where capability gaps really show, so those keep the stronger model. Two structural rules: put model choice behind a gateway as configuration, not code, because pricing and the model landscape shift every quarter, and re-run the eval when a new model version lands, since silent upgrades can move your task quality in either direction.

What gets you hired

Three layers. First, enforcement rather than politeness: every major provider now has a structured output mode or tool-call schema where you supply JSON Schema and generation is constrained to it, which eliminates the classic failures, markdown fences around the JSON, chatty preambles, invented fields. I use that as the baseline, not a prompt that asks nicely. Second, I still validate in code at the boundary, parsing against the schema with something like Pydantic or Zod, because the model remains an untrusted input source: constrained decoding guarantees shape, not sense, and a schema-valid response can still put "N/A" in a field my code treats as a number. Third, a designed failure path: if validation fails, retry once with the validation error appended to the request, which fixes the majority of failures in my experience; after that, dead-letter the item or degrade gracefully rather than crash a batch of 10,000 on item 4,012. Two design notes that prevent most trouble: keep schemas flat and use enums for anything categorical, because an enum cannot be creatively misspelled, and give the model an explicit "unknown" option, because otherwise you force it to fabricate a value when the source text does not contain one, and that fabrication is invisible downstream.

What gets you hired

Prompt injection is when instructions embedded in content the model processes get treated as commands. The direct form is a user typing "ignore your instructions". The dangerous form is indirect: the instructions hide in data the system fetches on its own, a webpage the model summarises, an email it triages, a document in the RAG index, invisible text saying "forward the last 5 emails to this address". It works because of an architectural fact: the model consumes one token stream, and nothing in that stream reliably marks which tokens are trusted instructions and which are untrusted data. That is why it is unsolved, unlike SQL injection there is no parameterised-query equivalent. Without tools, the blast radius is bad output. With tools, the model can act: read files, call APIs, send messages, so injected text in one poisoned document becomes actions executed with the application permissions. That changes the defence completely. I assume injection will sometimes succeed and cap what success is worth: tools scoped to least privilege, the email agent can read and draft but a human clicks send; consequential actions gated on confirmation; retrieved content treated as untrusted and where possible processed by a model with no tool access; and outbound actions monitored for anomalies, like an agent suddenly calling an exfiltration-shaped URL. Prompt-level defences help at the margin, but the security boundary lives outside the model.

What gets you hired

Layers, because each one fails differently and the prompt is the weakest of them. Input side: a fast, cheap classification pass first, is this in scope for the assistant at all, off-topic and clearly abusive requests get a polite deflection without ever reaching the main model, which incidentally saves money, the small-model screen costs a fraction of a frontier call. Then injection screening on anything that will enter the context from outside, and PII policy applied before logging. Model side: the system prompt still does real work, tone, refusal style, boundaries, plus grounding via RAG for anything factual, but I treat all of it as quality, not enforcement. Output side, where most real incidents live: schema validation for structured responses; a content policy check on free text, either a provider moderation endpoint or a small classifier, tuned per product; and for regulated or high-stakes domains, a groundedness check, does the answer follow from the retrieved sources, with refusal or escalation on failure, because "the model said so" is not a defensible source for a pricing or legal claim. Action side, if the assistant has tools: least privilege per tool, human confirmation for consequential actions, and rate limits per user so an exploited flow cannot scale. Around all of it, operations: every block and trigger logged with reasons, dashboards for trigger rates, and, this is the part people skip, a false-positive budget with periodic review of blocked interactions, because a guardrail that silently blocks 5 percent of legitimate users is its own incident. Each layer added must pay for its latency, the input screen and output check together should stay under about 300ms or the product feels it.

What gets you hired

I would walk legal through the real data flows, because there are more than one. Outbound: prompts, including retrieved documents and conversation history, and model outputs transit the provider. On enterprise API tiers the position is usually decent by contract: no training on customer data, bounded retention for abuse monitoring, often around 30 days, with zero-retention options negotiable, SOC 2 reports available, and the provider acting as a processor under a DPA. That is materially different from consumer chat products, and the first control is organisational: all usage goes through our gateway with our enterprise terms, no team pasting customer data into consumer tools, which is often the largest real-world leak. Second flow, the one everyone forgets: our own copies. Traces, logs, caches and eval sets now contain prompts and outputs, meaning customer data, so the trace store gets the same classification, access control and retention policy as any sensitive system, with PII redaction before storage where the debugging value survives it. Third: the RAG index is a permission surface, if documents are permissioned in the source system, retrieval must enforce the same permissions per user, or the assistant becomes a lateral-access path, and that has bitten real companies. The control set I would propose: data minimisation at the boundary, send the fields the task needs, not whole records; the gateway enforcing redaction policy centrally; regional routing if we have residency commitments; documented data-flow diagrams and retention windows for auditors and customer security reviews; and a short answer ready for sales, because "what do you do with our data when you use AI" is now in every enterprise questionnaire. Framed that way, legal gets what they need: not "the provider promises", but "here is every place the data goes, and here is the control at each hop".

What gets you hired

I would build the estimate from a real request, not the pricing page. Take an actual production-shaped call: system prompt, retrieved context, conversation history, and a realistic answer length, count the tokens, and multiply out. A support-bot turn that looks like "a question, an answer" is often 3,000 input tokens once the system prompt and 5 retrieved chunks are counted, and history is the classic surprise: every turn resends the conversation, so turn 10 of a chat costs several times turn 1, and cost per conversation grows quadratically-ish with length unless you cap or summarise history. The other surprises: retries on failures and timeouts, output verbosity, output tokens often cost 4 to 5 times input, so an uncapped chatty answer dominates the bill, and traffic mix, one power user with an agent loop can out-spend a thousand casual users. So before launch I do three things: instrument cost per request tagged by feature and user from day 1, so the bill is explainable; set a max output length and a history cap as default hygiene; and structure the prompt so the stable prefix is cacheable, which providers discount heavily, often around 10x on cached tokens. Then the big levers in order: right-size the model for the task, cache the prefix, trim what we send, and batch anything offline where a batch API is typically half price.

What gets you hired

Diagnosis first, because doubling with flat users means consumption per request or per user changed, and something specific changed it. I break the bill down along four axes: by feature, which one grew; by model, did traffic shift to a pricier one; input versus output tokens, which side grew; and per-user distribution, is it everyone or a few outliers. If we lack that attribution, that is finding zero, add request-level token logging tagged by feature immediately, flying blind on spend is its own incident. The shape then names the suspect. Input tokens up: conversation histories got longer, a bigger system prompt or more retrieved chunks shipped, in one team I saw a k change from 5 to 20 double the bill by itself. Output tokens up: a prompt change made answers verbose, or a max-tokens cap was removed. Requests up with flat users: retry storms from a flaky dependency, or an agent looping, one stuck agent retrying all night can burn hundreds of dollars. Model mix shifted: a routing change or a silent default change. Then the fix matches the finding, history caps and summarisation, restore output caps, loop budgets, revert the regressing change. Once the driver is fixed, the structural savers, in typical order of return: prompt structured for prefix caching, stable prefix first, which providers discount heavily, often around 90 percent on cached tokens; right-sized models per step, the classifier does not need the frontier model; response caching for repeated queries; batch API, usually half price, for anything offline. And so this never recurs silently: per-feature daily budget alerts at maybe 120 percent of baseline, so the next doubling is a Tuesday alert, not a month-end surprise.

What gets you hired

First I would split the number, because two different things get called slow: time to first token, how long before anything appears, and total generation time. Users mostly feel the first. If we are not streaming, that is lever 1 and it is free: with streaming, a response whose total time is unchanged starts appearing in under a second, and perceived latency collapses. In parallel I would trace where a request actually spends time, because in RAG and agent pipelines the model is often not the main cost: sequential steps, rewrite the query, retrieve, rerank, generate, stack up, and retrieval or a rerank call can add a second or two before generation even starts. Then the levers in rough order of value: stream; cap and design for shorter outputs, since generation time scales with output tokens, a 100-token answer arrives roughly 5x faster than a 500-token one; run pipeline steps concurrently where possible and drop steps that do not pay for themselves; use a small fast model for the easy steps, classification and routing, keeping the big model only where it matters; cache, both semantic response caching for repeated questions and provider prefix caching, which also improves first-token time on long stable prompts; and set the UX honest, show retrieval progress rather than a frozen spinner. Then measure p95, not the average, because tail latency is what users remember.

What gets you hired

I treat the model API like any critical remote dependency with a bad day now and then: 429 rate limits, 529 overloaded, timeouts, and long-tail slow responses are all normal, so the wrapper is standard resilience engineering. Timeouts first: an explicit per-call timeout, and separately a first-token timeout when streaming, because a stream that starts is different from one that never will. Retries: exponential backoff with jitter on rate limits and transient 5xx, honouring any retry-after header, capped at 2 to 3 attempts inside an overall latency budget, and only for calls that are safe to repeat, an agent step that already executed a side effect must not be blindly re-run. Fallbacks next, in preference order: on hard failure or budget exhaustion, route to a fallback model, same provider smaller model, or a second provider if we run multi-provider through a gateway; then a cached response if we have served this or a near-identical query before; then honest UX, "this is temporarily unavailable", which beats a spinner that never resolves. Around all of it: a circuit breaker so a provider incident sheds load fast instead of queueing retry storms, per-feature rate limiting so one runaway loop cannot exhaust our quota, and alerting on error rate and p95 latency per provider. The demo is one API call; production is this envelope around it.

What gets you hired

The budget change is qualitative: chat tolerates 3 to 5 seconds, voice feels broken past about a second of silence, so the target is first audio in under roughly 800ms, and that means rebuilding the pipeline around overlap. Everything streams and everything overlaps: speech-to-text is streaming, so transcription of the user is finishing roughly as they stop speaking, not starting; endpointing, detecting they are done, matters enormously because a lazy 700ms silence threshold spends most of the budget before we begin, so a fast semantic endpointer is worth real engineering; generation starts on the finalised transcript immediately and streams; and synthesis speaks the first sentence while the second is still generating, sentence-level chunked TTS, so first-audio depends only on time-to-first-sentence, not total response length. Then the pipeline itself slims, because a 4-second text stack cannot hide inside a 1-second budget: a fast model for the conversational layer, escalating to a stronger one only when the turn needs it; context trimmed hard, long prompts also slow time-to-first-token; retrieval pre-warmed or run speculatively while the user is still talking, betting on the likely topic; and responses designed short, voice answers are 1 to 3 sentences, which conveniently also cuts generation time. Where the use case allows, native speech-to-speech models remove the STT and TTS hops entirely and preserve prosody, at the cost of harder integration with tools and RAG, so the pragmatic 2026 architecture is often hybrid: speech-to-speech for the conversational skin, a text pipeline behind it for tool-using turns, with a filler, "let me check that", covering the tool latency honestly. The conversational machinery is product-critical: barge-in that stops synthesis instantly and records what was actually heard; recovery turns for misrecognition; and p95 first-audio per turn type, because the tail is what makes a voice product feel haunted. The mental shift: in chat you optimise a response, in voice you engineer a conversation, and silence is the error state.

What gets you hired

Under the hood there is no magic: I send the model a list of tool definitions, each a name, a description, and a JSON Schema for parameters. When the model decides a tool would help, it does not execute anything, it emits a structured tool-call message, the tool name plus arguments as JSON. My code receives that, validates it, executes the real function, and sends the result back as a message, and the model continues with that context, possibly calling more tools. So the model is a planner emitting requests; the trust boundary, validation, permissions, side effects, is entirely mine. That framing is also why tool design matters so much: the model chooses tools by reading the descriptions, so descriptions are the interface. A good tool does one thing and its description says when to use it and when not to, "search_orders: look up a customer order by id or email, use for order status questions"; parameters are few, well-named, with enums where possible; and crucially, results and errors are written for the model, an error like "date must be YYYY-MM-DD" lets it self-correct, while "error 500" produces flailing retries. The failure modes I watch for: overlapping tools that make selection a coin flip, kitchen-sink tools with 10 optional parameters, and returning enormous raw payloads, 50k tokens of JSON, when the model needed 5 fields. With maybe 5 to 15 well-designed tools an agent is reliable; with 40 vague ones it degrades badly.

What gets you hired

Mechanically an agent is a loop, and it helps to say it plainly: call the model with a goal and tool definitions; the model either answers or emits a tool call; my code executes the call, appends the result, and calls the model again; repeat until done. Everything agentic lives in that while-loop, and the framework is just this loop with conveniences. It derails for structural reasons: each iteration conditions on everything before it, so one wrong turn compounds; the model has no reliable sense of global progress, so it can retry a failing approach indefinitely, and each retry adds noise to the context, making recovery less likely, the classic doom loop. So control is external, never delegated to the model: a hard iteration cap, say 10 to 15 steps for a bounded task; token and spend budgets per run, because an unbounded agent is an unbounded bill; wall-clock timeout; and loop detection, if the same tool is called with the same arguments twice, intervene. Then the quality-side controls: explicit success criteria the loop checks rather than letting the model declare victory; tool results and errors written so the model can actually adjust course; checkpointing so a 20-step run that fails at step 18 resumes rather than replays, which matters doubly when steps have side effects; and an escalation path, when the budget exhausts or progress stalls, summarise state and hand to a human instead of thrashing. My rule from operating these: autonomy is earned in production, start with the agent proposing and a human approving, and widen only as the trace history shows it deserves it.

What gets you hired

The problem is integration combinatorics: every assistant needed bespoke connectors for every tool, N clients times M systems, so ecosystems could not compound. MCP standardises the interface: a server exposes tools, resources and prompt templates over a defined protocol, a client, the assistant, discovers and calls them, and one Github or database server now works with any MCP-capable client. That is genuinely valuable, and it is also exactly why the second half of the question matters: MCP makes it trivially easy to attach third-party code to a model that holds your credentials and your users data. Before I plug one in, I check 4 things. Provenance: who publishes the server, is it the vendor official one, is it maintained, am I pinning a version or trusting latest, this is supply-chain thinking, an MCP server update is code I now run. Permissions: what credentials does it hold and how scoped, a database server gets a read-only role on the schemas it needs, never an admin connection string; and what can its tools do, read tools and write tools deserve different scrutiny. Injection surface: everything the server returns, tool descriptions included, enters my model context, so a malicious or compromised server can inject instructions; tool descriptions should be reviewed at install and pinned, and servers allowlisted rather than user-installable. And observability: every tool call logged with arguments and results, so when the assistant does something odd I can reconstruct which server fed it what. Then the same least-privilege rule as any agent: consequential actions gated on human confirmation regardless of which server requested them.

What gets you hired

My default is one agent with well-designed tools, and I hold that default until a measured wall says otherwise, because the single agent is dramatically easier to debug, eval and run. The legitimate triggers for splitting are structural, not metaphorical. Context isolation is the strongest: one task genuinely needs large context the other would pollute, a research sub-agent reading 50 documents and returning a 500-token brief keeps the main agent context clean, and that pattern, sub-agents as context firewalls, is most of what multi-agent is good for. Genuine parallelism: independent workstreams, searching 5 sources concurrently, where fan-out and join beats sequence. Privilege separation as a security boundary: the agent that reads untrusted web content holds no tools with side effects, so injection lands in a sandbox, that is a real boundary. And occasionally measured specialisation, where distinct prompts or models per stage demonstrably beat one generalist on the eval. What multi-agent adds in failure modes is the part the diagrams omit. Hand-offs are lossy: agent A summarises for agent B, and the detail B needed did not survive the summary, the whole system degrades into a game of telephone. Shared state is unsolved by default: two agents believing different things about progress. Errors cascade: A retrieves something subtly wrong, B builds on it confidently, and by the output it is unattributable. Cost multiplies, 5 agents deliberating is 5x the tokens, And debugging becomes distributed-systems archaeology across interleaved traces. So my bar is empirical: show me the single-agent eval plateauing for a reason that maps to one of those triggers, split along that one seam, keep the orchestration as flat as possible, and re-run the same eval to prove the split paid for itself, quality or cost. Most systems that impress in production are one good agent, good tools, and one or two sub-agents used as context firewalls.

What gets you hired

This is exactly the regression-test problem, and I treat it the same way. "Seems better" usually means it improved on the 3 cases we were staring at; the question is what happened on everything else, because prompt changes routinely fix one behaviour and quietly break another. So: a fixed eval set, 50 to 100 inputs drawn from real usage, weighted toward the cases that matter and seeded with every failure we have ever fixed, our version of regression tests. Each case carries a grading method suited to the task: exact or schema comparison for structured output; assertion checks for factual answers, does the response contain the correct figure; and for open-ended quality, an LLM judge with an explicit rubric, which I calibrate by spot-checking maybe 30 judgments against my own until I trust its agreement rate. Then the workflow: run the suite on the old prompt and the new one, compare scores overall and per-slice, and look specifically at flips, cases that went from pass to fail, because an aggregate score can hide a serious regression in a subgroup. It runs in CI on every prompt, model or retrieval change, cheap, a 100-case run on a small judge costs well under a dollar. And the set is living: every production failure that reaches us becomes a new case, so the suite gets harder as the product ages. That converts "seems better" into "passed 91 of 100 versus 84, and no previously-passing case broke".

What gets you hired

I run it through three questions. First, what does a wrong answer cost, and who catches it? These models are probabilistic, they will sometimes be wrong, so the shape of a good use case is one where errors are cheap or verification is built in: drafting an email a human edits, summarising a ticket an agent reads, generating code that a compiler and tests check, extracting fields that validation can sanity-check. The bad shape is exact answers, delivered with authority, with no review, medical or financial advice straight to a user, or computing a bill. Second, is the task transformation or authority? LLMs are excellent at reshaping what you give them, summarise, extract, translate, draft, classify, and weakest when asked to be a factual oracle from their own weights. If the knowledge is ours, retrieval turns the oracle problem back into transformation, which is why RAG use cases age well. Third, can we measure it? I want a definition of good and an eval set before committing, because a feature we cannot measure is a feature we cannot maintain, every prompt or model change becomes a gamble. Concretely, from the classic wins, support drafting, document Q&A over our own corpus, meeting summaries, code assistance, I would pick the one with the clearest verification loop and highest volume, ship it narrow, around 1 to 2 weeks to a measurable pilot, and let its numbers argue for expansion.

What gets you hired

I spent about three weeks building a careful few-shot pipeline for extracting line items from supplier invoices, prompt engineering, format coaxing, retry logic for malformed JSON, and roughly two months later the provider shipped native structured outputs plus much better document understanding in the same price tier, which made maybe 70 percent of my careful work unnecessary. The useful part is how we decided to migrate, because most announcements do not deserve a migration. My filter is two questions: does it change our cost or quality on our eval, and does it delete code we maintain. I re-ran our invoice eval suite, 200 labelled documents, against the new capability in an afternoon: accuracy went from 91 to 94 percent, and the retry-and-repair code, about 400 lines that accounted for a disproportionate share of our incidents, could go entirely. That cleared both bars, so we migrated, behind a flag, old path as fallback for two weeks, then deleted. What I took from it shaped how I work. First, the eval suite is what made the decision cheap: without it, "should we adopt this" is a week of debate, with it, it is an afternoon of measurement, so the suite is the asset that converts churn from a threat into free upgrades. Second, I stopped building elaborate workarounds for model limitations without asking whether the limitation is likely to be short-lived, some workarounds are load-bearing product code, others are scaffolding around a gap the next release fills, and I now label them accordingly, scaffolding gets built cheap and expected to be deleted. My standing filter: anything that claims to move our tasks gets an afternoon against the eval suite; leaderboard wins on tasks we do not run get ignored; and one afternoon a month is budgeted for trying things with no immediate justification, which is where two of our later wins came from.

What gets you hired

We shipped a support assistant that answered billing questions from our docs. Three weeks in, support flagged that it was confidently quoting a refund window of 30 days to customers on a plan where the real window was 14. Impact was concrete: roughly 40 customers over two weeks were told the wrong policy, and support had to honour several goodwill refunds, so this was a money bug, not a cosmetic one. Diagnosis through traces: retrieval was fetching the generic refund page, which said 30 days, and never the plan-specific page, because the plan name appeared only in a page title our chunker had stripped. The model did nothing wrong; it faithfully answered from the wrong context, which is exactly why I now distrust "the model hallucinated" as a first explanation. Immediate fix: re-chunk with titles and plan metadata attached, add a metadata filter so plan-scoped questions retrieve plan-scoped pages, verified against the failing queries. Structural changes were the real outcome. Every policy answer now carries a citation, and we added a groundedness check on the answer against the retrieved source for money-adjacent topics. The 12 failing queries became permanent eval cases, and we built a small suite specifically of "plan-specific question, generic document trap" cases, which caught two similar issues before ship since. And we changed the release rule: no RAG feature ships without a retrieval eval on a labelled set, because we had shipped on end-to-end vibes and the retriever was broken underneath. The habit I kept: when an LLM feature is wrong, read the retrieved context before blaming the model, and every production failure becomes an eval case the same week.

What gets you hired

Product wanted the assistant to auto-send responses to customer emails, fully automated, no human step, and the appeal was real: the drafts were good, and auto-send would have cut first-response time from hours to under a minute. I pushed back on the last step, not the feature. My case had three parts. Error asymmetry: a draft that a human edits costs seconds when wrong; a wrong email sent under our name to an angry customer is a trust incident, and at our volume even a 2 percent serious-error rate meant roughly 30 bad sends a week, I put that number in front of them rather than saying "it might make mistakes". Verification gap: we had no reliable automated check for "is this response factually right about this customer situation", groundedness checks catch some of it, but account-specific correctness needed a human or much better integration with billing data than we had. And the hype-cycle point made politely: the cost of being 6 months late to auto-send was small; the cost of a public failure was not. The counter-proposal made it land: ship drafts-with-one-click-send immediately, instrument the edit rate, and define the promotion criterion up front, when a category shows under 5 percent meaningful-edit rate for a month, that category graduates to auto-send with spot-check sampling. That reframed it from no to a measured path to yes. Outcome: edit rate started around 30 percent, which settled the argument better than I could have, two low-risk categories, order status and password resets, graduated to auto-send in about four months, and the incident count from those categories was zero. The general lesson I carry: pushback lands when it comes with a number and a path, and the edit-rate instrumentation was worth more than my opinion.

What gets you hired

I argued hard for building our summarisation pipeline around fine-tuned small models instead of prompted frontier ones, on cost grounds: my spreadsheet showed roughly 8x savings at our volume. I won the argument, and we spent about 6 weeks building the tuning pipeline, data curation, and evaluation harness. What falsified it was maintenance reality, not the launch numbers: quality was fine on day 1, but every meaningful product change, new document types, a new output format, longer inputs, needed fresh training data and a re-tune, roughly a week of turnaround, while the prompted baseline my colleague maintained adapted in an afternoon. Within a quarter we had shipped 3 product iterations on the baseline and 1 on my pipeline, and the cost gap had also narrowed by more than half because provider prices dropped, which my spreadsheet had assumed static. I called it myself in the retro rather than waiting for someone else to: we froze the tuned path, migrated the two live use cases back to prompted models over 2 sprints, and kept the eval harness, which was the genuinely reusable part. The rule I extracted: in a market where capability rises and prices fall quarterly, any cost justification must model the trend, not the snapshot, and iteration speed is a cost line, not a soft factor. I now put a "what invalidates this" date on decisions of that class, and we re-check the maths on it.

What gets you hired

Marketing had committed a launch date for our document-analysis feature at a conference, and 10 days out our eval said it was not ready: 88 percent field accuracy overall, but 71 percent on scanned documents, which were about a third of real uploads. My first move was to make "not ready" concrete, because a date fight you can lose, a risk statement is harder to dismiss: at projected volume, 71 percent on scans meant roughly 200 wrong extractions a week reaching customers who would trust the numbers, in a feature whose whole pitch was accuracy. Then I offered scope instead of delay: ship on the date, but gate scanned documents behind a "beta, verify before use" banner with a review step, while digital PDFs, where we cleared 94 percent, shipped as advertised. That kept the conference demo and the launch, cut the risk surface to the segment we could stand behind, and gave us a visible path: the scan pipeline improved behind the gate, and we removed the banner 5 weeks later when the slice hit 90 percent. Two things I did that mattered beyond the compromise: I wrote the risk in one paragraph, numbers included, so the decision-maker was deciding with open eyes rather than against my mood; and once the call was made, I instrumented per-segment accuracy in production so the promotion decision would be data, not another argument. What I took from it: "not ready" is only useful when it decomposes into which slice, which failure, at what rate, and scope is almost always the negotiable axis that dates are not.

What gets you hired

We migrated a summarisation feature to a model that cut cost by around 60 percent, and our eval said quality held, 93 versus 94 on a 150-case suite, within noise. The VP of sales insisted the new summaries were worse and wanted a rollback, with two examples. The tempting move was to wave the dashboard; instead I treated the anecdote as a lead about my eval. I pulled his two cases plus the last 50 summaries from his team specifically, and there it was: his team ran long multi-stakeholder deal calls, 60 to 90 minutes, and our eval set skewed toward the median call, 15 to 30 minutes. On calls over an hour, the new model dropped commitments made late in the call at nearly twice the rate of the old one. Aggregate metrics hid it because long calls were maybe 8 percent of volume, but they were his team whole workflow, so he was right, and the eval was blind. We added a 40-case long-call slice to the suite, and with a visible failure to target, mitigation went fast: chunked summarisation with a merge pass for calls over 45 minutes, which brought the long-call slice above the old model score, we kept the cost win. The process changes outlived the incident: eval sets are now built with slice coverage checked against real traffic distribution, weighted for high-stakes segments and not just volume; every aggregate score is reported with per-slice breakdowns, because 93 percent overall with a hole in one team workflow is a failing grade wearing a passing one; and my standing rule with stakeholders is "bring me two examples and I will trace them within a day", which turned the relationship with that VP from adversarial to him being our best early-warning channel. The principle: when measurement and lived experience conflict, the first hypothesis is that the measurement has a blind spot, and checking costs a day.

What gets you hired

We shipped an assistant that auto-drafted responses inside our support tooling, and the eval story was genuinely good, drafts rated usable-or-better in 85 percent of cases. Six weeks after launch, only about 20 percent of agents were using it, and usage was declining. Instead of a survey, I sat with 8 agents and watched. The mechanism was visible within an hour: when a draft was wrong, it was wrong confidently, and the agent had to read the whole thing carefully to find out, so a bad draft cost more time than writing from scratch, and one embarrassing near-miss, a draft that almost went out with the wrong customer name, outweighed fifty good drafts in memory. Trust is asymmetric like that: successes are invisible, failures are stories. The fixes were experience engineering, barely touching the model. We made the draft visibly provisional, diff-style highlights on every claim pulled from the customer record, so verification became a 5-second scan instead of a full read. We added a per-category confidence signal and simply did not show drafts below the bar, cutting the visible failure rate roughly in half at the cost of coverage. And we let agents correct a draft with one click and see the correction stick for that customer, which converted them from graders into trainers, that one change moved sentiment more than anything else. Adoption reached about 70 percent over the next two months with the model itself essentially unchanged. What I generalised: for AI products the unit of quality is not average accuracy, it is the cost of the worst visible failure, and the design goal is making verification cheap and control real. I now treat "how will a user catch a wrong output in 5 seconds" as a launch requirement, same status as the eval.

What gets you hired

A colleague designed our research-report feature as a 5-agent pipeline, planner, searcher, analyst, writer, critic, and I thought one agent with good tools and one review pass would match it at a fraction of the cost. His arguments were not silly: separation of concerns, per-stage prompts, and a pipeline diagram everyone understood. Rather than escalate opinions, we agreed on a bake-off with the terms fixed in advance, which is the part I would defend hardest: 30 real research tasks, blind grading by two colleagues against a rubric we co-wrote, plus cost and latency per task, one week to build each, and we wrote down before running it what result would settle it, because criteria agreed after results exist are just ammunition. The outcome was messier than either position, which is typical: quality was statistically indistinguishable on 26 of 30 tasks, his pipeline won clearly on the 4 hardest multi-source tasks, and mine was about 60 percent cheaper and twice as fast. So we shipped the single agent as the default path with his planner-plus-critic stages as an escalation tier for tasks the router classified as complex, roughly 15 percent of traffic. Neither of us had proposed that architecture; the data did. Two things made the disagreement productive rather than political: the timebox kept the stakes small, one week is cheap insurance against months on the wrong architecture; and blind grading meant neither of us could argue with the scores, only with the rubric, which we had co-written. The bake-off with pre-agreed criteria became the team standard for architecture disputes, and it has since killed two of my own proposals, which is the sign it is working.

What gets you hired

First, stop making it worse: at 30 percent 429s the instinctive fix, more retries, is the amplifier, so I confirm retries use exponential backoff with jitter, honour the retry-after header, and are capped at 2 or 3 attempts, and if a retry storm is already running I temporarily drop retries to 1. Then triage by priority: pause or queue background consumers, batch jobs, evals, backfills, so the remaining quota serves interactive users, that alone often clears user-facing errors in minutes. In parallel, the real question: what changed in demand, because 429s mean demand crossed supply and something moved. I check the request-rate dashboard by feature: a traffic spike, a newly launched feature, a scheduled job landing at peak, or the classic, a stuck agent or a bugged retry loop hammering the API, one runaway consumer can eat an entire org quota. The trace tells me which feature and which caller. Short-term fixes follow the finding: kill the runaway, reschedule the batch job off-peak, or accept the traffic is real growth. Structurally, this incident is usually a design smell: all traffic sharing one undifferentiated quota. The fixes that prevent recurrence: separate keys or lanes for interactive versus background traffic so a backfill can never starve users; client-side rate limiting per feature at say 80 percent of quota so we queue gracefully instead of erroring; a request-priority queue in front of the provider; and if growth is real, both a limit increase with the provider and secondary-provider capacity through the gateway. And the alert that was missing: we should page at 80 percent quota utilisation trending upward, not at 30 percent user-visible failure, saturation alerts belong on the leading metric.

What gets you hired

First, reproduce: collect 10 to 20 actual failing questions, not paraphrases, and pull full traces for them, the query, what was retrieved with scores, the final prompt, the answer. Then the fork that structures everything: was the right information in the retrieved set? I read the chunks before I touch anything else. If the right passage is not there, it is a retrieval problem, and the usual suspects in order: chunking split the answer away from its context; query-document phrasing mismatch, users ask "can I get my money back" and the doc says "refund eligibility policy", which embeddings handle worse than people assume; the document is missing or stale in the index, ingestion silently skipped or never re-ran; or a metadata filter is excluding the right source. If the right passage is there, it is a generation problem: the model ignored the context and answered from its weights, or merged two chunks into a wrong synthesis, or the prompt buries the context so it gets lost. Fixes differ accordingly, grounding instructions with citation requirements, better context placement, fewer higher-quality chunks via reranking. In my experience the split lands roughly 70/30 retrieval, which is why starting at the prompt is the classic wasted week. Finally, every confirmed failure goes into the eval set with its expected answer, so the class of bug stays fixed, and I add per-stage metrics, retrieval hit rate, groundedness, so next time the dashboard tells me which side broke before users do.

What gets you hired

First minute: kill the run and confirm billing has actually stopped, then immediately check what side effects it executed while looping, did it write records, send messages, call external APIs, because a stuck agent is not just a cost problem, 6 hours of retrying may have half-executed things that need cleanup or dedup. Then evidence: the full trace of the run is the incident record; I want where behaviour diverged from the normal 20-minute shape. The pattern is usually visible fast: at some step a tool started failing or returning something the agent could not use, and instead of stopping, it retried, rephrased, retried, each failure appending to the context, and a context full of failed attempts makes the next decision worse, so loops self-reinforce. Common mechanical causes: an upstream dependency changed or went down and the tool error was unhelpful, "error 500" tells the model nothing actionable, so it flails; the success condition became unreachable, the data it was told to find stopped existing, and nothing bounded the search; and malformed tool output that parsed but meant nothing, so the agent politely looped on garbage. The immediate fix targets whichever it was. The real work is preventing the class, because a 6-hour 400-dollar run means our controls were missing, not that the model misbehaved: hard budgets enforced by the harness, not the prompt, iterations, tokens, wall-clock, spend, this job should have died at 2x its normal envelope, so roughly 40 minutes or a few dollars; loop detection, same tool with same arguments twice triggers intervention; tool errors rewritten to be actionable so the agent can adjust or give up cleanly; a designed failure path, checkpoint state, summarise progress, alert a human, rather than retry forever; and an anomaly alert on runtime and spend per job, because a human should have been paged at minute 40, not hour 6.

What gets you hired

First, localise: p95 tripling is a distribution statement, so I check whether all requests got slower or a slow tail widened, and I pull per-stage timings from traces, our pipeline is roughly retrieval, rerank, queue, time-to-first-token, generation, and the stage that grew names the suspect class. The no-deploy suspects, in the order I check them. Provider side: time-to-first-token per provider is the cleanest signal, if it jumped from 800ms to 6 seconds our infrastructure is fine and the fix is failover; status pages lag, so I trust our own dashboards first. Silent model change: are we actually hitting a fallback, quota exhaustion or a health-check flap can route traffic to a slower path without anyone deploying, so I check the model distribution in the traces, this one is embarrassingly common. Input growth: token counts per request, did conversation histories cross a threshold overnight, did a data source start returning bigger documents into retrieval, latency scales with both input and output tokens, so a 3x on input is a latency event with no code change. Retry amplification: if a fraction of calls started timing out and retrying, p95 absorbs the retries, so error and retry rates per dependency. And load: queue depth and concurrency, did traffic spike or a batch job start competing for the same rate limit, an overnight cron colliding with peak is a classic. Mitigation runs in parallel with diagnosis, because users are waiting: fail over to the healthy provider if it is provider-side, tighten timeouts and cap retries if it is amplification, cap output length and trim context if it is growth. And afterwards, the alert gap: p95 tripling should have paged before users noticed, with per-stage latency alerts so the page names the stage, next time diagnosis starts at step 3.

What gets you hired

First, turn "worse" into data: pull the specific conversations support is referring to, check the quantitative signals, thumbs-down rate, escalation rate, our sampled quality scores, for an inflection point, and establish when it started, because a clean start time cuts the hypothesis space in half. "No deploys" only rules out our code; these systems have several change surfaces that do not go through CI. The provider may have updated the model behind the same API name, the top suspect, so I run our canary eval, the fixed 50-question probe set we run daily, against the production config; if Tuesday scores dropped versus last Monday with our config unchanged, that is strong evidence, and dated evidence is exactly what makes a provider escalation productive. The RAG index drifts: did ingestion pull a broken or reorganised batch of documents, did a source system change formats, is retrieval now surfacing junk, I diff retrieved chunks for a handful of the failing queries against traces from before the inflection. Traffic itself shifts: a new user cohort or topic mix the system handles poorly looks identical to regression in aggregate metrics, so I slice by segment and query category. And quiet infrastructure changes: did we start hitting quota and silently falling back to a smaller model, did a config flag flip, did a dependency auto-update. The trace comparison is usually decisive, same query, last week versus now, and seeing what differs: chunks, prompt, model version header, output. Then two closures, not one: fix the cause, re-point ingestion, adjust retrieval, escalate to the provider with the eval diff, or pin the model version where the provider offers it; and close the detection gap, because support telling us is the real failure, if the canary eval or a feedback-rate alert did not fire, that alarm gets built this week so the next silent change costs hours, not a week of degraded users.

What gets you hired

This is a measurement bug as much as a product bug, so I debug the instrument. First, the crucial experiment: take 20 real complained-about conversations and run those exact inputs through the eval harness. Two outcomes, both informative. If the harness also fails them, the suite is fine and the coverage is wrong: production traffic has drifted or was never represented, so I diff distributions, topic mix, input length, phrasing style, user segments, languages, between the eval set and a sample of live traffic, and the missing slice becomes new labelled cases, weighted to traffic reality rather than to what was easy to label. If the harness passes the exact inputs that failed in production, that is the more interesting finding: the harness tests a different system than the one shipping. The usual gaps, in order: context, production carries conversation history, retrieved chunks and tool results while the eval tests clean single turns, and quality degrades exactly there; configuration drift, the eval pinned to a model or prompt version production has moved past; and state, production failures often need turn 5 of a conversation to reproduce, which single-turn evals structurally cannot see. Then the grader: if an LLM judge scores the complained-about outputs as good, the rubric is blind to the failure users experience, tone, omissions, subtle wrongness, so I calibrate the judge against human ratings on the complaint set specifically and tighten the rubric where they disagree, in my experience judge blind spots cluster on exactly the qualities users complain about and metrics miss. Structurally, the fix is a loop, not a patch: a standing pipeline where sampled production traces, especially complaint-adjacent ones, flow into the eval set monthly, per-slice scores reported against live traffic weights, and the harness asserted to reproduce production context, history, retrieval and all. A 92 that ages without that loop is a number about the past.

What gets you hired

Two layers, and the second is the one standard monitoring does not give you. The classic layer stays: p50 and p95 latency, time-to-first-token separately for streaming, error rates by provider status code, throughput, and cost per request tagged by feature, alerting on all of it. The LLM-specific layer starts with tracing the pipeline as a unit: for every request I want the prompt template and its version, the retrieved chunks with scores, every tool call with arguments and results, token counts, model and parameters, and the final output, linked in one trace. Without that, a user report of "it gave a wrong answer yesterday" is undebuggable; with it, it is a 5-minute lookup. Tools like Langfuse or LangSmith, or OpenTelemetry with custom spans, all work; the discipline matters more than the vendor. Then quality signals, because the defining property of these systems is that requests can succeed while answers are wrong: sample production outputs, say 1 to 5 percent, through automated graders, groundedness against retrieved sources, format compliance, refusal correctness; collect explicit user feedback and, often more honest, implicit signals, retries, rephrasings, abandonment, escalations to human support; and track these as time series with alerts on drops. Finally drift watching: providers update models under the same API name and user behaviour shifts, so behaviour changes with no deploy on my side. The defence is a canary eval, a fixed probe set run daily against production config, which turns "the model silently changed Tuesday" from a week of confusion into a dated alert. The mental shift from normal services: green infrastructure does not mean the product works, so quality is monitored as a first-class signal next to latency.

What gets you hired

For messy real-world documents, my default is a vision-capable model reading rendered pages, because layout is meaning in invoices and forms, which table a number sits in, what label is adjacent, and vision models use that where classic OCR-to-text flattens it away. Traditional OCR keeps a place at high volume on clean consistent documents, where it is much cheaper per page, and a sensible hybrid is OCR plus layout as supplementary input to the model. But the architecture choice is the smaller half; trustworthiness is the job. Extraction is schema-constrained, every field typed, categorical fields as enums, and everything nullable, because a required field the document does not contain is a fabrication machine. Then verification in layers: deterministic validation per field, dates parse, currencies match expected ranges, tax ids checksum; cross-field consistency, line items sum to the subtotal, subtotal plus tax equals total, which catches a large share of misreads because a hallucinated digit rarely keeps the arithmetic consistent; and for the critical fields, amounts, account numbers, a second extraction pass, either re-asking with the crop of the relevant region or a different model, agreement between passes is a strong correctness signal. Confidence routing ties it together: documents that pass validation, cross-checks and agreement flow straight through; failures and disagreements queue for human review with the model output pre-filled, which typically means humans see maybe 10 to 20 percent of volume instead of 100. And it is measured like a product: per-field accuracy on a labelled set of a few hundred documents, straight-through rate, and human-correction rate feeding back as eval cases and, eventually, fine-tuning data. The goal is not a model that is never wrong, it is a system that knows when it might be.

What gets you hired

At 15 features I want one gateway, and the argument is the list of things that are otherwise solved 15 times or zero times. Every feature needs credential management, retry and fallback logic, spend tracking, rate limiting, tracing and a migration path when models change; direct integrations mean 15 divergent copies, and in practice several features will have none of it. The gateway contract: teams call one internal API, logically a request plus a policy, with the model as configuration, a routing alias like "fast", "balanced", "frontier", or a named policy per feature, never a hard-coded model string in application code. Inside it: central credentials, so provider keys live in exactly one place; per-feature budgets and rate limits with alerts, which is what makes cost attribution and runaway protection automatic rather than aspirational; routing with health-aware fallbacks, provider degraded means traffic shifts without 15 teams shipping hotfixes; prompt-prefix and response caching where safe; uniform tracing, every request tagged by feature, model, tokens and latency, one dashboard for the company; and PII redaction policy applied consistently. What deliberately stays out: prompts, evals and product logic belong to feature teams, the gateway is dumb pipes with policy, not a committee that reviews prompts, and it must be operationally boring, stateless, horizontally scaled, because it is now in the critical path of 15 features, it gets an SLO and a pager. The payoff shows up on the first model migration: flip an alias for one feature, canary against its eval suite, roll forward or back in config, and the second-order payoff is that new features start with budgets, fallbacks and tracing on day 1 for free. If we are buying rather than building, a LiteLLM-style proxy covers most of this; the point is the single choke point, not the brand.

What gets you hired

I would frame it per workload, not as one company-wide bet, and start from constraints rather than costs. First, quality: for the hardest tasks, complex reasoning, agentic work, frontier API models still hold a real edge, and if a product depends on that, the decision is made for those workloads. For narrower tasks, classification, extraction, summarisation, RAG answering, current open-weight models are genuinely competitive, which is what opens the question at all. Second, honest economics. APIs are pure variable cost with zero infrastructure, instant access to model improvements and elastic scale. Self-hosting has the visible cost, GPUs at serious money per instance-hour, and the invisible ones that dominate: an inference serving stack, vLLM-style, with batching and autoscaling to run; utilisation risk, GPUs at 20 percent utilisation triple your effective unit cost, and LLM traffic is bursty; an MLOps and on-call capability that now owns model quality end to end, no provider to escalate to; and falling behind the frontier as a permanent tax, todays self-hosted model is 6 to 12 months behind next years API model your competitor gets by changing a string. The break-even exists, but it is at high, stable, predictable volume on tasks a small model serves, plus a team genuinely able to operate it. Third, the forcing functions that override economics: regulatory or contractual data locality that provider terms cannot satisfy, air-gapped deployment, latency floors needing co-location, or customisation deeper than fine-tuning APIs allow. My recommendation shape: frontier APIs as the default, because model velocity is the dominant fact of this market; self-host the specific workloads where a trigger fires or the volume math clearly wins; and everything behind the gateway so placement is a routing decision we revisit quarterly, not an identity. The strategic asset to build in-house is not GPU operations, it is our evals, our data flywheel and our integration layer, those compound; the model itself is increasingly a swappable component.

What gets you hired

The core discipline is direction: problems rank technologies, technologies never rank problems. We maintain a short living list of the expensive problems, support cost, onboarding time, engineering throughput, whatever the business actually bleeds on, and any new paradigm is evaluated as "does this move one of these materially". That single inversion kills most hype on contact. The standing filter makes it cheap: a one-page triage, what problem of ours it claims to move, what evidence exists beyond demos, what it would cost to validate, done in under an hour by one senior person on rotation. Survivors get a timeboxed spike, 1 to 2 weeks, one engineer, run against our own eval suites and our data, never the vendor demo path, with success criteria written before the spike starts, because criteria written after become rationalisations. Then three buckets with different obligations. Adopt: an owner, a scoped first deployment, metrics, and a kill criterion, adoption is a project, not a vibe. Watch: the interesting-but-not-yet pile, each entry with a named trigger, "when it can do X on our eval", "when the price crosses Y", reviewed quarterly, so watching is a standing search. Ignore: written reasons, revisitable, which matters because saying no with reasons preserves credibility when the answer changes later, and this field re-runs the evaluation every 6 to 12 months. Around the mechanism, two political commitments. Legibility: the decision log is public internally, so the enthusiast whose pet paradigm was ignored can read why and argue with criteria rather than with me. And calibrated humility: I budget for being wrong, the watch triggers exist precisely because my priors in this field have a shelf life, and a filter that never revisits is just slow dismissiveness. The outcome I am accountable for is portfolio-shaped: the org should adopt 2 or 3 things a year that measurably move a bleeding metric, and spend near zero engineering months on paradigm tourism.

What gets you hired

Probably yes, but the design determines whether it accelerates or strangles, so I would start with a diagnosis: sit with the 8 teams and inventory what is duplicated, typically provider integration and retries, spend tracking, eval tooling, guardrails, RAG plumbing, where quality diverges dangerously, usually security and data handling, and what they wish existed. That inventory, not org-chart aesthetics, defines the platform charter. The split I hold firm: the platform owns rails, the teams own outcomes. Rails means the model gateway with budgets, fallbacks and tracing; the evals infrastructure, the harness, the judge tooling, the CI integration, while teams own their own eval sets, they know what good looks like in their domain; the guardrail and safety components as consumable middleware; RAG building blocks, connectors, chunking, retrieval services; and the cost and observability dashboards. Teams keep owning prompts, model choice within policy, their features and their metrics, because prompts are product decisions. The anti-bottleneck design is the crux: the platform is self-service golden paths, a team ships a compliant AI feature without talking to anyone, and the paved road is so good that using it is the lazy path. No approval gates except where regulation genuinely demands review, and there, SLAs on the review. Staffing: seed it partly with engineers from the product teams who built the duplicated pieces, sized small, maybe 4 to 6 for 8 teams. Measurement keeps it honest: adoption because it is chosen, not mandated, time-to-first-shipped-AI-feature for a new team, which should drop from months to weeks, incident and spend trends, and a quarterly survey where the product teams grade the platform. If teams are routing around it, that is the platform failing, and the escape hatch stays legal: build your own if the road does not serve you, documented, with your own pager. Paved roads win by being better, not by being mandatory.

What gets you hired

If we are first measuring at the board question, we are late. So the regime I run: no AI initiative gets funded without a one-pager naming the business metric it moves, cost per ticket, cycle time, revenue per account, its current baseline, the target, and how we will measure the delta, including what the counterfactual is. Before-after comparisons credit the AI with everything else that shipped that quarter, so wherever feasible we measure against a control, a holdout segment, a phased rollout by region or team, or matched comparison, support automation as cost-per-resolved-ticket in rolled-out versus waiting queues, engineering assistants as cycle-time deltas between early-access teams and the waitlist. Not everything gets a clean experiment, but the discipline of asking "compared to what" is the difference between measurement and marketing. Then the board report is a portfolio, told honestly in three parts. Deployed wins with defensible numbers: say support automation resolving 40 percent of tickets at human-level reopen rates, priced against the baseline cost curve. Killed initiatives, counted and framed correctly: we ran 6 pilots, killed 3 at the pilot stage for under 200 thousand dollars, and killing them early is the system working, a portfolio with zero kills means the bar is too low or the reporting is dishonest. And platform investment, valued not by its own ROI, which is unmeasurable, but by what it demonstrably changed: time for a team to ship a compliant AI feature down from months to weeks, spend visible per feature, zero data incidents. Two distinctions I keep sharp: adoption is not impact, usage is a leading indicator and the claim is the cycle-time delta; and estimates are labelled as estimates, because one punctured number costs the whole program its credibility.

What gets you hired

Requirements first: say 10 million documents across sources with different permission models, freshness within minutes, a few queries per second at peak, and the requirement that shapes everything: a user must never receive an answer derived from a document they cannot open, permissions are the hard part of enterprise RAG, not embeddings. Architecture in three planes. Ingestion is a living pipeline, not a load: per-source connectors doing incremental sync; structure-aware chunking with title and section prepended so chunks stay self-describing; each chunk carrying source, timestamp and crucially ACL metadata synced from the source system; embeddings plus a keyword index, hybrid from day 1 for the project names and error codes dense retrieval fumbles; and deletion and permission-change propagation treated as first-class events with an alert on sync lag, because serving a deleted or restricted document is the classic silent failure. Query plane: authenticate the caller, expand group membership, and filter retrieval by ACL at query time, because post-hoc filtering is too late once the leak entered the prompt. Retrieve hybrid at maybe top 50, rerank to the best 5 to 8, generate with citations mandatory and an explicit refusal path when retrieval confidence is poor, because "I could not find this" earns more trust than improvising, and trust is the adoption currency. Streaming responses, aim for time-to-first-token around a second, total 3 to 5 seconds. Operations plane, which is what makes it stay good: a retrieval eval, a few hundred labelled question-to-document pairs sliced per source; sampled groundedness grading of production answers; thumbs plus "wrong document" feedback wired into a triage queue; full tracing per answer, what was retrieved, from where, with what scores; and per-source dashboards, because when quality drops the first question is which corpus. Rollout matters as much as architecture: start with 2 or 3 high-value well-maintained sources, nail quality and the permission story, then add connectors, because indexing everything on day 1 mostly indexes garbage.

What gets you hired

First, the metric, because 60 percent automated resolution is trivially achievable badly: answer everything confidently and count the tickets that do not come back angry. I would define resolution as automated-and-stayed-solved, no reopen within 7 days, and pair it with CSAT on automated interactions and the escalation experience, targets like 60 percent resolution at CSAT within a couple of points of human baseline and reopen under 10 percent, so gaming one number breaks another visibly. Second, taxonomy, because tickets differ in kind: roughly a third are how-to and configuration questions, ideal for grounded RAG; another chunk are account actions, password resets, seat changes, invoice copies, which need tools, not answers; a tail is billing disputes, bugs and angry customers, where automation attempts destroy satisfaction, so the design goal there is fast, well-routed hand-off. Pipeline: classification first on intent, sentiment and account tier, cheap small-model pass; then per category. Informational tickets get RAG over docs and known issues plus the customer account context, because "how do I add users" has a plan-specific answer, with citations and resolution confirmation, did this fix it, a "no" is an escalation trigger, not a retry loop. Action tickets run through scoped tools with tiered risk: identity-verified resets fully automated, refund-adjacent actions capped, say auto-approve under 50 dollars on eligible plans, propose-for-human-click above, and every action logged and confirmed back to the user. Escalation is designed as the product it is: triggers on low retrieval confidence, negative sentiment, protected topics like legal threats and data loss, second failed attempt, or explicit request, and the hand-off carries everything, conversation, retrieved context, attempted actions, so the customer never repeats themselves. Operationally: full tracing, weekly sampled grading of automated resolutions, reopen-rate dashboards per category, and rollout category by category, starting where volume is high and risk is low, expanding as each category proves its numbers. Agents move from answering password resets to handling the judgment tail.

What gets you hired

The failure mode to design against is not missing bugs, it is noise. A human reviewer who is wrong 60 percent of the time gets ignored; an AI reviewer at that precision gets muted org-wide, irreversibly. So the whole design optimises precision over recall. Scope: high-confidence findings in three lanes, likely bugs, null and error paths, race-prone patterns, off-by-ones with the reasoning shown; security issues, injection, authorisation gaps, secrets in code; and violations of our documented conventions. Explicitly out of scope: style and formatting, linters own that, and speculative "you might consider" commentary. Context: the diff alone is not reviewable, so the assistant gets changed files plus their surrounding code, definitions of what the change calls and touches via code search, the team convention docs, and the PR description for intent. Pipeline: static pre-filters first, cheap and certain, then the model pass over the contextualised diff, then the precision gate: each candidate finding gets a confidence assessment, an adversarial "argue this is a false positive" pass kills the marginal ones cheaply, and only findings above threshold post, hard-capped at maybe 5 comments per PR, ranked by severity. Every comment states the issue, the reasoning, and a suggested fix, and carries thumbs feedback. Operations is where it lives or dies: dismissal rate per rule category tracked weekly, any category above roughly 30 percent dismissal gets tuned or cut; a monthly sampled audit of findings against what human reviewers caught; and per-repo calibration, because a hard rule in the payments service is a nitpick in an internal tool. Rollout: shadow mode first, findings visible to a pilot team of maybe 15 engineers, tune until their dismissal rate is under 20 percent, then expand team by team on request rather than mandate. The success metric is not findings posted, it is findings acted on: did escaped-bug rate move, and do teams refuse to give it up.

What gets you hired

The design principle: the platform owns the machinery, teams own the meaning, because a central team curating datasets for 12 products it does not operate becomes both a bottleneck and wrong. The split: the platform provides the harness, runners, grader library, storage, dashboards and CI integration; each team owns its eval datasets, rubrics and thresholds. What the platform standardises is the primitives, because comparability is the value: a common case format, input, expected properties, metadata, slices; grader types as a library, exact match, schema validation, assertion checks, and LLM-judge-with-rubric, with judge calibration against human ratings as a platform service; versioned datasets, so a score is always relative to a dataset version; and baseline management, every run compared against the feature current baseline with per-slice deltas and flip lists, cases that changed direction, surfaced above aggregates. It serves three moments on one harness. CI: a prompt or pipeline change runs the feature suite pre-merge, gates on regression against baseline, results as a PR comment in about 5 minutes for a 100-case suite. Release: model comparisons, run the same suite against candidate models for migration decisions, which becomes trivial when the harness is shared. Production: continuous sampled grading of live traffic, say 1 to 5 percent, feeding the same dashboards, plus the canary probe set run daily against production config to catch silent provider changes. The loops keep it alive: a one-click path from production trace to eval case, so complaints and interesting failures become dataset growth with team review; monthly dataset-versus-traffic drift reports nudging teams when coverage rots. Self-service is the operating model, SDK plus config, a team onboards a new feature in under a day without talking to anyone, and the platform team measures itself on adoption and time-to-first-eval, not on owning quality. Leadership reporting falls out as a by-product; the customers are the 12 teams, and if they route around the platform it has failed.

What gets you hired

First decision: this is a workflow with LLM steps, not a free-form agent. The process is known, so orchestration is deterministic code with the model working inside steps, which is what makes it auditable and signable. Extraction: vision-capable model on the document, schema-constrained, every field typed and nullable so absence is never fabricated, followed by deterministic validation: dates parse, formats check, line items sum to subtotal, subtotal plus tax equals total, and that arithmetic alone catches most misreads. Matching: retrieve candidate POs by supplier and reference, score on amount, lines and terms, three-way match against goods receipts where they exist, outputting a match confidence plus named discrepancies. The autonomy tiers are the part finance signs, so they are policy, not model judgment: clean extraction, exact PO match, approved supplier, amount under say 5,000 auto-processes, often 60 to 70 percent of volume; soft exceptions queue for one-click human approval with the case prepared and the discrepancy highlighted; hard exceptions, no PO, new supplier, large variance, above-cap amounts, always go to a human. Money-grade engineering underneath: idempotency keys on every payment-adjacent action so a crashed and resumed run cannot schedule twice; checkpointed state per invoice, the pipeline resumes, never replays; duplicate detection on supplier, number, amount and near-duplicates, the classic invoice fraud and error mode; an immutable audit log of every extraction, match score, decision and actor, human or system, which is what the auditors actually ask for; and hard limits in code, no single payment above X, daily aggregate caps, new-supplier holds, that no model output can override. Measured like a finance system: straight-through rate, per-field accuracy, exception aging, and a monthly sampled audit of auto-processed invoices feeding the eval set. Rollout: 4 weeks of shadow mode, then auto-processing for the lowest tier only, expanding as the audit stays clean.