Stop reading whenever you have enough

Fault injection, in twelve questions.

Each one is the question the previous answer makes you ask. The ones that matter come with the manifest or the command, because that is the part you meet at work.

Not technical? The first three questions are written for you, and they are enough to follow any conversation about this.

Rungs
12
Hands on
4
Read
~8 min
01

What it is

01–03

You can make a service pretend to be slow or broken, on purpose, for one tester, to see what happens next.

three minutes, no cluster needed · for anyone

  1. 01

    Why break something on purpose?

    Not knowing this costs

    Every resilience feature you never tested is a guess in production.

    • Your timeouts and fallbacks exist as untested code
    • Their first real test is otherwise a real outage
    • A drill moves that test to a Tuesday afternoon
  2. 02

    What faults can the mesh manufacture?

    Not knowing this costs

    Slow and broken are different failures, and systems fail them differently.

    • A delay: the answer arrives late
    • An abort: the answer is an error code
    • Both injected by the proxy, upstream untouched
  3. 03

    Who should experience the fault?

    Not knowing this costs

    An ungated fault is an outage you wrote yourself, with your name on it.

    • One tester carrying one header, usually
    • A tiny percentage during a planned game day, rarely
    • Everyone, never

Before you scroll on

0/3

You should now be able to

02

The YAML you will see

04–05

A delay or an error, a percentage, and a header that keeps it away from real users.

the two fault types and the gate around them · for whoever writes the drill

  1. 04

    What does an injected delay look like?

    hands on

    Not knowing this costs

    A fault smaller than the timeout proves nothing and reassures everyone.

    • A fault block with fixedDelay and a percentage
    • Gated behind a match on a drill header
    • Sized above the timeout you want to prove
    manifesttwo seconds, for one person
      http:
    - match:
        - headers:
            x-chaos-drill:
              exact: latency-2s
      fault:
        delay:
          fixedDelay: 2s
          percentage:
            value: 100
      route:
        - destination:
            host: payment-svc.payments.svc.cluster.local
            subset: stable
    - route:                    # everyone else, untouched
        - destination:
            host: payment-svc.payments.svc.cluster.local
            subset: stable
    The delay is above the caller 1s perTryTimeout on purpose: the drill proves the timeout, so the fault must be big enough to trigger it.
  2. 05

    What does an injected error look like?

    hands on

    Not knowing this costs

    Fallback code that has never seen a 503 usually throws on its first.

    • abort with an HTTP status instead of delay
    • It tests the fallback path, not the timeout
    • The caller should render its degraded answer
    manifesta manufactured 503
          fault:
        abort:
          httpStatus: 503
          percentage:
            value: 100
    Aborts return instantly, so they exercise error handling without the wait. Use the delay to test patience and the abort to test the apology.

Before you scroll on

0/3

You should now be able to

03

Proving it works

06–08

The drill has a hypothesis: the timeout fires, the retry recovers, the user never knows.

the drill, run end to end · for whoever signs off the resilience story

  1. 06

    What is the hypothesis, exactly?

    Not knowing this costs

    A drill without a hypothesis produces vibes, not evidence.

    • Written before the drill, falsifiable, one line
    • Fault fires, timeout catches, retry lands on a healthy pod
    • The end user metric does not move
  2. 07

    Did the fault actually arrive?

    hands on

    Not knowing this costs

    A drill that never verified the gate may be running on real users.

    • Send the header, time the response
    • Without the header, time it again
    shellthe fault, measured
    kubectl exec deploy/sleep -n payments -- curl -s -o /dev/null \
  -H "x-chaos-drill: latency-2s" \
  -w 'with header: %{time_total}s\n' checkout:8080/api/summary

kubectl exec deploy/sleep -n payments -- curl -s -o /dev/null \
  -w 'without:     %{time_total}s\n' checkout:8080/api/summary

# with header: 2.01s   <- fault arriving
# without:     0.04s   <- gate holding
    Both numbers matter: the first proves the fault, the second proves the gate. Skipping the second is how ungated faults ship. The prerequisite nobody checks: the fault is on the route into payment-svc, so checkout has to forward x-chaos-drill on its outgoing call. If it drops the header, both numbers come back identical and the drill silently proved nothing.
  3. 08

    Did the protection fire, or just the fault?

    hands on

    Not knowing this costs

    Proving the fault without proving the catch is testing the wrong thing.

    • The timeout counter must move on the caller
    • The user facing request must still succeed
    • Fault fired plus user fine equals the system worked
    shellthe other half of the proof
    kubectl exec deploy/checkout -c istio-proxy -n checkout -- \
  pilot-agent request GET stats | grep -E 'rq_timeout|retry_success'

# upstream_rq_timeout: 3        <- timeout caught the delay
# upstream_rq_retry_success: 3  <- retry rescued each one
    If the fault fired and these counters stayed flat, the delay flowed straight through to the user and your timeout is not configured where you think it is.

Before you scroll on

0/3

You should now be able to

04

When it breaks

09–10

The drill itself can become the incident, and it has, everywhere.

the ways drills go wrong · for whoever runs game days

  1. 09

    How does a drill become an incident?

    Not knowing this costs

    An injected fault left overnight is tomorrow’s mystery latency ticket.

    • The gate is forgotten and the fault hits everyone
    • Or the percentage outlives the game day
    • Faults are config: they persist until deleted
  2. 10

    Why did 50 percent abort produce 75 percent errors?

    Not knowing this costs

    Percentage math that ignores retries misreads every drill result.

    • Retries re roll the dice per attempt
    • Two attempts at 50 percent fail together only a quarter of the time
    • Injected faults and retries interact, by design

Before you scroll on

0/3

You should now be able to

05

Where it ends

11–12

Injected faults are clean. Real failures are not.

the limits, where people get caught · for whoever plans chaos engineering

  1. 11

    What can injection not simulate?

    Not knowing this costs

    Passing clean drills builds confidence real failures do not honour.

    • Partial brownouts: slow for some keys, fine for others
    • Resource exhaustion, corrupted responses, clock skew
    • The mesh injects clean failures; reality is dirtier
  2. 12

    Is passing the drill the same as being resilient?

    Not knowing this costs

    The outage that gets you is the one no drill was written for.

    • No. It proves the paths you thought to test
    • Game days add the human half: alerts, runbooks, handoffs
    • The drill is the floor, not the ceiling

Before you scroll on

0/3

You should now be able to

Go deeper

3 links, each earning its place.

Fault injection task↗

The official runnable drill, delay and abort both, on the bookinfo sample.

HTTPFaultInjection reference↗

Every field of the fault block, including the percentage semantics from rung 10.

Principles of chaos engineering↗

The hypothesis discipline from rung 6, as the discipline it came from.

Where this leaves you

Rung 4 is the manifest. Rungs 7 and 8 are the two measurements that make it a drill instead of a stunt.

If you keep one thing: the fault is not the test. The catch is the test, and the fault is just how you summon it.