Stop reading whenever you have enough

Resilience, in thirteen questions.

Each one is the question the previous answer makes you ask. The ones that matter come with the manifest or the command, because that is the part you meet at work.

Not technical? The first three questions are written for you, and they are enough to follow any conversation about this.

Rungs
13
Hands on
6
Read
~9 min
01

What it is

01–03

When a service you depend on struggles, you can wait, try again, or stop asking. These are those three choices, written down.

three minutes, no cluster needed · for anyone

  1. 01

    What problem is this solving?

    Not knowing this costs

    Most outages are one slow service dragging its callers down with it.

    • A dependency slows down or starts failing
    • Untreated, its problem becomes your problem
    • Three bounded reactions replace unbounded waiting
  2. 02

    What are the three, in one line each?

    Not knowing this costs

    Every resilience conversation is these three, in some disguise.

    • Timeout: how long you will wait
    • Retry: how many times you will re ask
    • Breaker: when you stop asking entirely
  3. 03

    Why must they agree with each other?

    Not knowing this costs

    Tuned separately, the three protections attack the same upstream together.

    • Retries multiply load exactly when load is the problem
    • Timeouts decide how fast retries fire
    • The breaker caps what the other two can amplify

Before you scroll on

0/3

You should now be able to

02

The YAML you will see

04–06

Timeouts and retries live on the route. Breakers live on the destination. One incident, two files.

the three knobs and where each lives · for whoever writes the manifests

  1. 04

    How do the timeout numbers add up?

    hands on

    Not knowing this costs

    A retry that cannot fit its budget is pure wasted upstream load.

    • perTryTimeout times attempts must fit inside timeout
    • The route timeout is the promise to your caller
    • Their timeout must be bigger than your whole budget
    manifesta budget that adds up
    # VirtualService route
      timeout: 3s              # total promise to the caller
      retries:
        attempts: 2
        perTryTimeout: 1s      # 1s + 1s + 1s <= 3s, it fits
        retryOn: connect-failure,refused-stream,unavailable
    Three tries at one second fit a three second budget. Set perTryTimeout to 2s and the second retry can never complete: it is cancelled mid flight by the route timeout, every time.
  2. 05

    Which failures are safe to retry?

    hands on

    Not knowing this costs

    retryOn: 5xx on a checkout route is a duplicate charge policy.

    • Connection failures: the request never arrived
    • Explicit overload signals like refused-stream
    • Never blanket 5xx on routes that change things
    manifestthe safe list, and the trap
          retries:
        attempts: 2
        retryOn: connect-failure,refused-stream,unavailable
        # NOT retryOn: 5xx on a write route:
        # a 500 after the work is done means the work happens twice
    connect-failure is safe because the upstream never saw the request. A 5xx is ambiguous: the work may have completed before the error. Ambiguity plus retry equals duplicates.
  3. 06

    Where does the breaker live and what trips it?

    hands on

    Not knowing this costs

    A limit copied from a blog post is sized for someone else’s traffic.

    • connectionPool caps concurrency, outlierDetection ejects failers
    • Both in the DestinationRule, both per proxy
    • Size from observed peak, then add headroom
    manifestboth halves of the breaker
    # DestinationRule
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 32
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 34
    Limits are enforced per caller proxy, not per service. Ten caller pods with maxConnections 100 can open a thousand connections between them.

Before you scroll on

0/3

You should now be able to

03

Proving it works

07–09

Each protection has a counter. If the counter never moves, the protection was never tested.

commands that answer yes or no · for whoever has to sign it off

  1. 07

    Are the protections actually firing?

    hands on

    Not knowing this costs

    A protection first exercised during a real incident is a hypothesis.

    • Each one owns a counter on the caller proxy
    • Flat counters mean untested, not unnecessary
    shellthe three, on one screen
    kubectl exec deploy/checkout -c istio-proxy -n checkout -- \
  pilot-agent request GET stats | grep -E 'timeout|retry|overflow' | grep payment

# upstream_rq_timeout: 12              <- timeouts firing
# upstream_rq_retry_success: 9         <- retries rescuing
# upstream_rq_pending_overflow: 0      <- breaker not yet needed
    Read these during a load test, not just in an incident. The counters are the only proof the YAML you wrote maps to behaviour you get.
  2. 08

    Can I trip the breaker on purpose?

    hands on

    Not knowing this costs

    The drill costs ten minutes. Discovering the rule bound to nothing costs an incident.

    • Drive concurrency past the pool with a load tool
    • Watch UO appear and overflow counters climb
    • This is a ten minute test in a dev cluster
    shellthe ten minute drill
    # squeeze the pool first: the pending limit is what actually overflows
#   connectionPool:
#     tcp:  { maxConnections: 1 }
#     http: { http1MaxPendingRequests: 1, maxRequestsPerConnection: 1 }

kubectl exec deploy/fortio -n test -- \
  fortio load -c 3 -qps 0 -n 30 http://payment-svc.payments:8080/health

# Code 200 : 11    Code 503 : 19    <- the breaker refusing
# 503 UO upstream_reset_before_response_started{overflow}
    maxConnections alone will not do this: excess requests queue instead of overflowing, so the drill looks like nothing happened. It is http1MaxPendingRequests that turns a queue into a 503 UO. Confirm with upstream_rq_pending_overflow from rung 7.
  3. 09

    Which protection fired? The log knows.

    Not knowing this costs

    Three flags, three different fixes. Guessing picks the wrong one twice.

    • UT: the route timeout gave up
    • URX: the retry budget ran out
    • UO: the breaker refused before sending

Before you scroll on

0/3

You should now be able to

04

When it breaks

10–11

Resilience features cause their own incidents, and those are the embarrassing ones.

the failures you will actually hit · for whoever gets paged

  1. 10

    The upstream got slow and total traffic tripled. What is it?

    hands on

    Not knowing this costs

    The instinctive fix during a retry storm makes the storm worse.

    • A retry storm: every failure re asked twice
    • The fix is fewer retries, not more capacity
    • Retries against total requests is the amplification
    shellamplification, measured
    kubectl exec deploy/checkout -c istio-proxy -n checkout -- \
  pilot-agent request GET stats | grep -E 'rq_retry|rq_total' | grep payment

# upstream_rq_total: 14203
# upstream_rq_retry: 9871          <- ~70% of load is retries
# upstream_rq_retry_overflow: 0    <- nothing is capping them
    retry_overflow only moves if connectionPool.http.maxRetries is set, and Istio leaves it effectively unlimited, so a default install shows zero here in the middle of a storm. That zero is the finding: nothing is capping the amplification. Raising attempts in response pours fuel on it.
  2. 11

    The breaker trips at half the configured limit. Why?

    Not knowing this costs

    A pool sized at the service level is wrong the day the callers scale.

    • The limit is per proxy and you scaled the callers
    • Or the pool serves several routes you forgot share it
    • Count concurrency per caller pod, not per service

Before you scroll on

0/3

You should now be able to

05

Where it ends

12–13

These bound how a failure spreads. They cannot make a failing service succeed.

the limits, where people get caught · for whoever reviews the design

  1. 12

    What does the mesh still not do?

    Not knowing this costs

    A perfect breaker in front of an app with no fallback is a fast error page.

    • Decide what to show a user when the call fails
    • Make a write safe to repeat
    • Both are application code, and always were
  2. 13

    Where does the shed load actually go?

    Not knowing this costs

    A breaker with no plan for the refused traffic just moves the outage upstream.

    • Refused requests surface as errors somewhere
    • Something above must degrade gracefully: cache, queue, or apology
    • Breakers relocate pain, they do not delete it

Before you scroll on

0/3

You should now be able to

Go deeper

4 links, each earning its place.

Circuit breaking task↗

The official version of rung 8: tripping the pool with fortio and reading the counters.

Envoy circuit breaking model↗

What the thresholds actually gate underneath, including the per priority detail Istio hides.

Envoy retry semantics↗

Every retryOn value from rung 5, with the exact conditions each one matches.

Envoy response flags↗

The UT, URX and UO table behind rung 9.

Where this leaves you

Rung 4 is the arithmetic most configs get wrong. Rungs 7 and 8 are the proof that yours is not one of them.

If you keep one thing: retries multiply load exactly when load is the problem. Every other sentence on this page is a footnote to that one.