LLM Cost & Latency Optimization

Your demo worked. The bill and the p95 did not.

The demo was magic. You wired up a model, pasted in a fat system prompt, stuffed the whole knowledge base into the context, and it answered beautifully. Then you shipped it to real traffic. Two things broke at once: the monthly bill climbed past what the feature is worth, and the p95 latency, the slow-tail experience your impatient users actually feel, crept toward six, eight, ten seconds. Suddenly the magic feels like a liability.

Here is the good news: almost none of this requires a smarter model. It requires *engineering*. The same disciplines you already apply to databases and APIs, caching, routing, trimming, batching, map cleanly onto LLM calls. This article is the playbook: the levers that make an LLM app cheap and fast enough to keep in production, in the order you should reach for them.

The principle: don't send the token in the first place

The cheapest, fastest token is the one you never send.

Every optimization here is a variation on that idea. You pay, in dollars *and* in milliseconds, roughly in proportion to the tokens that flow in and out of the model. Input tokens cost money to send and add to the time-to-first-token. Output tokens cost more per unit *and* are generated one at a time, so they dominate latency. So the entire game is: send fewer tokens, generate fewer tokens, and avoid the call entirely when you can.

Crucially, optimizing cost and optimizing latency are not always the same move. Some levers buy you both (a cache hit is free *and* instant). Some buy one at the expense of the other (batching is cheaper but slower per request). Knowing which is which is what lets you tune deliberately instead of flailing.

The picture: a request's journey to the cheapest answer

Before reaching for any single model, a well-built LLM service runs each request through a funnel. Every stage is a chance to answer *cheaper* or *not at all*. Only the requests that survive every filter reach the expensive large model.

1. 1

   Check the semantic cache

   Embed the incoming request and look for a near-identical question answered before. On a hit, return the stored answer instantly, zero model tokens, single-digit-millisecond latency.

2. 2

   Route on a cache miss

   A lightweight classifier (rules, a tiny model, or a heuristic) decides whether this is an easy request or a hard one. Most production traffic is easy.

3. 3

   Try the small model first

   Send easy requests to a small, cheap, fast model. It handles the long tail of simple lookups, classifications, and short rewrites for a fraction of the cost.

4. 4

   Escalate only when needed

   Hard requests, or small-model answers that fail a confidence/validation check, escalate to the large model. You pay the premium only for the requests that truly need it.

5. 5

   Stream the answer back

   Whichever model answers, stream tokens to the client as they generate. Total time is unchanged, but perceived latency collapses because the user sees words immediately.

6. 6

   Write back to the cache

   Store the answer keyed by the request embedding so the next similar question skips the whole funnel.

The levers, and what each one actually buys you

These are the six moves worth knowing, ranked roughly by impact-per-effort. The honest framing matters: each lever has a trade-off, and stacking them blindly can hurt. Read the table as a menu, not a checklist.

A minimal router and token-budget trimmer

Here is a stripped-down version of the funnel as code: a token-budget trimmer that keeps context under a hard cap, and a router that tries a small model first and escalates only when the answer looks weak. The point is the *shape*, swap in your own client, models, and validation.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def token_len(text: str) -> int:
    return len(enc.encode(text))

def trim_to_budget(chunks: list[str], budget: int) -> list[str]:
    """Keep the most relevant chunks first; drop the rest once the budget is spent."""
    kept, used = [], 0
    for chunk in chunks:  # chunks arrive pre-sorted by relevance
        cost = token_len(chunk)
        if used + cost > budget:
            break  # the cheapest token is the one you never send
        kept.append(chunk)
        used += cost
    return kept

SMALL, LARGE = "small-fast-model", "large-strong-model"

def looks_weak(answer: str) -> bool:
    """Cheap heuristics to decide if we must escalate."""
    flags = ("i'm not sure", "i cannot", "as an ai")
    return len(answer) < 20 or any(f in answer.lower() for f in flags)

def answer(question: str, context_chunks: list[str], client) -> str:
    context = "\n\n".join(trim_to_budget(context_chunks, budget=1500))
    prompt = f"Context:\n{context}\n\nQuestion: {question}"

    # 1) try the small model first
    draft = client.complete(model=SMALL, prompt=prompt, max_tokens=300)

    # 2) escalate only if the cheap answer is weak
    if looks_weak(draft):
        return client.complete(model=LARGE, prompt=prompt, max_tokens=300)
    return draft

Two details earn their keep. First, trim_to_budget enforces a hard input ceiling, without it, a few oversized retrieved chunks silently double your bill on every call. Second, looks_weak is deliberately crude; in production you'd validate against the task (did it return valid JSON? did it cite a source?) rather than sniffing for phrases. The architecture is identical to what you'd build for an AI agent: a cheap default path with a deliberate escalation rule.

Common mistakes that quietly drain the budget

1. Always using the biggest model. The flagship is the most expensive and the slowest. Most production traffic is easy and a small model handles it fine, you just never measured the split. Route first; reserve the big model for the requests that earn it.
2. No caching of any kind. Real traffic is repetitive: the same questions, the same system prompt, the same retrieved chunks. Without exact, prompt, or semantic caching you re-pay for identical work all day. Caching is usually the single highest-leverage change you can ship.
3. Ignoring output tokens. Teams obsess over shrinking the prompt and forget that output tokens cost more *and* dominate latency. A max_tokens cap and a prompt that says "answer in two sentences" often saves more than any input trimming.
4. Letting context grow unbounded. Stuffing the whole knowledge base "just in case" inflates every single call. Retrieve and trim to a token budget; relevance beats volume.
5. Optimizing latency you can't feel. Chasing total generation time when the real fix is *perceived* latency. Stream the response and the same model feels twice as fast.
6. Tuning blind. Shipping levers without per-request cost and p50/p95 latency dashboards. If you can't see the bill and the tail, you can't tell whether a change helped or hurt.

Takeaways

Where to go next

Cost and latency optimization is the final discipline that turns an LLM demo into a shippable product. The strongest next step is to wire a real dashboard, per-request cost, cache hit rate, and the small-vs-large routing split, so every lever you add is measured, not guessed.

• Solidify the fundamentals these levers rest on with How LLMs Actually Work, why tokens cost what they cost.
• See the routing-and-escalation pattern in a fuller system in Building AI Agents.
• Put it all together along the AI Engineer career path, where cost and latency sit alongside evaluation and reliability.