Stop reading whenever you have enough

Troubleshooting, in twelve questions.

Each one is the question the previous answer makes you ask. Nearly all come with the command, because this whole page is the part you meet at work.

Not technical? This page is the most technical in the track. Read rungs 1 and 2 for the shape of it, then send it to whoever is on call.

Rungs
12
Hands on
6
Read
~9 min
01

The method

01–02

Mesh debugging is a fixed walk through five layers. Panic is skipping layers.

the fixed order, before any commands · for anyone who will ever be on call

  1. 01

    What is the walk?

    Not knowing this costs

    Every skipped layer is an assumption, and assumptions are where hours go.

    • Config valid? Fleet synced? Pod configured?
    • Then: what did the proxy log, and why
    • Five layers, always in that order
  2. 02

    Is it the mesh or the app? One split, first.

    Not knowing this costs

    This one field routes the incident to the right team in minute one.

    • Read the response flag on the failure
    • A dash means the app answered: not the mesh
    • Any letters mean the proxy decided something

Before you scroll on

0/3

You should now be able to

02

The first five minutes

03–05

Validate the config, check the fleet, ask the pod. In that order, every time.

the three commands that start every incident · for whoever gets paged

  1. 03

    Is the config even coherent?

    hands on

    Not knowing this costs

    Sixty seconds of analyze regularly replaces an afternoon of log reading.

    • analyze cross checks resources against each other
    • It catches what single file review cannot
    shellthe linter for the whole mesh
    istioctl analyze -A

# IST0101 Referenced host+subset not found: payment-svc v2
# IST0108 Unknown annotation ...
# clean output = move to the next layer, config is coherent
    The dangling subset, the unbound gateway, the short host: the classics from every other ladder surface here first, before any traffic evidence.
  2. 04

    Did the config actually reach the proxies?

    hands on

    Not knowing this costs

    Debugging behaviour against config a proxy never received is pure loss.

    • proxy-status, the fleet in one screen
    • Anything not SYNCED is serving old rules
    shellthe fleet check
    istioctl proxy-status | grep -v 'SYNCED.*SYNCED.*SYNCED.*SYNCED'

# empty  -> everyone has current config, move on
# rows   -> those pods run stale config: your change may not be live
    My change did nothing has two causes: it never applied, or it never distributed. analyze catches the first, this catches the second.
  3. 05

    What does this one workload actually have in force?

    hands on

    Not knowing this costs

    Applied and in force are different facts, and only one of them pages you.

    • describe merges every policy touching the pod
    • Routes, mTLS mode, authz policies, one view
    shellone pod, whole truth
    istioctl x describe pod payment-svc-7d9c-x2k4 -n payments

# VirtualService: payment-svc (weights 99/1)
# Workload mTLS mode: STRICT
# Authorization policies: deny-all, payment-svc-callers
    The gap between what you applied and what this prints is where namespace scoping, selectors and precedence quietly ate your intent.

Before you scroll on

0/3

You should now be able to

03

Reading the evidence

06–07

The proxy wrote down what happened and why. Most incidents end at reading it.

flags, logs and programmed state · for whoever owns the diagnosis

  1. 06

    What is the flag telling me to do next?

    hands on

    Not knowing this costs

    The flag is the index into every other ladder. Use it as one.

    • UH or UO: read clusters and endpoints
    • NR: read routes. UF: read the upstream pod
    • UT or URX: read timeouts and retries
    shellflag first, then the matching command
    kubectl logs deploy/checkout -c istio-proxy -n checkout --tail=50 \
  | awk '{print $6}' | sort | uniq -c | sort -rn

#  312 -      <- app answered; not a mesh problem
#   41 UH     <- go read: proxy-config cluster / endpoint
#    3 UT     <- go read: the route timeout
    Counting flags across recent traffic beats staring at single lines: the distribution says whether you have one incident or three.
  2. 07

    The flag points at config. How do I see what is programmed?

    hands on

    Not knowing this costs

    The walk converts what is wrong into which file is wrong.

    • Walk the chain: listeners, routes, clusters, endpoints
    • Stop at the first output that surprises you
    shellthe four stage walk, compressed
    istioctl proxy-config routes   deploy/checkout -n checkout --name 8080
istioctl proxy-config cluster  deploy/checkout -n checkout --fqdn payment-svc.payments.svc.cluster.local
istioctl proxy-config endpoint deploy/checkout -n checkout \
  --cluster "outbound|8080||payment-svc.payments.svc.cluster.local"
    This is the Envoy ladder’s walk, deployed under pressure. The surprise stage names the broken resource: route means VirtualService, cluster means DestinationRule, endpoint means labels or health.

Before you scroll on

0/3

You should now be able to

04

The classics

08–09

Five patterns cover most mesh pages. Recognising them is most of the job.

the five incidents everyone eventually has · for whoever wants the shortcuts

  1. 08

    Which five patterns cover most pages?

    Not knowing this costs

    Five patterns, five ladders, most of a year of incidents.

    • Instant 503 NR: dangling subset. UH: labels match nothing
    • 503 UF milliseconds, one dependency: STRICT versus DISABLE
    • Waves on a timer: ejection. UO under load: pool. Silent policy: scope
  2. 09

    What confirms each classic in one command?

    Not knowing this costs

    Recognition plus one confirming command is the whole senior on call act.

    • Subsets: proxy-config cluster, look for the missing row
    • mTLS clash: upstream proxy log, TLS error HTTP_REQUEST
    • Ejection: stats grep outlier. Pool: stats grep overflow

Before you scroll on

0/3

You should now be able to

05

Where it ends

10–12

Half of mesh tickets are not mesh problems. Proving that quickly is a skill.

when it is not the mesh · for whoever closes the ticket

  1. 10

    How do I prove it is the app, kindly?

    hands on

    Not knowing this costs

    Blame with evidence lands as help. Blame without it starts a war.

    • Flags are dash, duration is the app thinking
    • The proxy delivered and waited; the app answered slowly or badly
    • Hand over the log line, not the accusation
    shellthe polite proof
    kubectl logs deploy/payment-svc -c istio-proxy -n payments --tail=5 \
  | awk '{print "code=" $5 " flag=" $6 " duration=" $12 " upstream=" $13}'

# code=500 flag=- duration=4890 upstream=4887

# flag "-", duration ~= upstream time:
# the mesh delivered in 3ms; the app spent 4.9s deciding to fail
    Field 12 minus field 13 is the mesh’s share of the latency. When that gap is milliseconds, the remaining seconds belong to the application. Same positional fields as the flag count in rung 7.
  2. 11

    What goes in the escalation?

    Not knowing this costs

    A good escalation is the walk, written down. It halves the next person’s time.

    • The flag distribution, the describe output, the walk result
    • What you ruled out, in walk order
    • Three artefacts beat thirty screenshots
  3. 12

    What do I write down afterwards?

    Not knowing this costs

    An incident you did not write down will be investigated again from scratch.

    • The symptom to classic mapping you just learned
    • Into the runbook, beside these commands
    • Next time it is recognition, not investigation

Before you scroll on

0/3

You should now be able to

Go deeper

4 links, each earning its place.

Diagnostic tools overview↗

Every command on this page, in its official home.

istioctl analyze codes↗

What each IST code from rung 3 means and how to fix it.

Envoy response flags↗

The complete flag table behind rungs 2 and 6.

Common problems↗

The project’s own list of classics, overlapping rung 8 and extending it.

Where this leaves you

Rungs 3 to 5 are the first five minutes of every incident. Rung 6 is the index that routes everything after.

If you keep one thing: config valid, fleet synced, pod configured, flag read, chain walked. In that order, even when the dashboard is red and someone is standing behind you.