Stop reading whenever you have enough

Envoy, in twelve questions.

Each one is the question the previous answer makes you ask. The ones that matter come with the command, because that is the part you meet at work.

Not technical? The first three questions are written for you, and they are enough to follow any conversation about this.

Rungs
12
Hands on
3
Read
~8 min
01

What it is

01–03

The helper beside every program is a real piece of software with a name. This is that software.

three minutes, no cluster needed · for anyone

  1. 01

    What is Envoy, in one sentence?

    Not knowing this costs

    Every Istio feature you configure is Envoy behaviour underneath.

    • A proxy: software that forwards network traffic
    • Built at Lyft for exactly this job
    • The second container in every meshed pod
  2. 02

    Why did Istio not build its own?

    Not knowing this costs

    The mesh only works because proxies never restart to learn new rules.

    • Envoy already existed, battle tested at scale
    • Its defining trick: reconfigurable while running
    • Istio is a control plane wrapped around that trick
  3. 03

    What does reconfigurable while running mean?

    Not knowing this costs

    Without this, every VirtualService edit would be a rolling restart.

    • New routes and clusters arrive over an API
    • Connections in flight are never dropped for config
    • That API family is called xDS

Before you scroll on

0/3

You should now be able to

02

The chain inside

04–06

A request crosses four stages: a door, a decision, a group, an address. Every mystery lives at one of them.

the four stages every request crosses · for whoever will ever debug a proxy

  1. 04

    What happens to one request inside the proxy?

    Not knowing this costs

    Four stages, in a fixed order. Every debugging session walks them.

    • Listener: the door the traffic entered
    • Route: which destination this request matches
    • Cluster, then endpoint: the group, then the address
  2. 05

    Where do the Istio resources land on the chain?

    Not knowing this costs

    This mapping is why istioctl has exactly the subcommands it has.

    • VirtualService becomes routes
    • DestinationRule becomes cluster policy and subsets
    • Service endpoints become the endpoint list
  3. 06

    What are LDS, RDS, CDS and EDS?

    Not knowing this costs

    A proxy can be SYNCED on three channels and stale on the one that matters.

    • One delivery channel per stage of the chain
    • The columns in istioctl proxy-status, exactly
    • SYNCED per column means that stage is current

Before you scroll on

0/3

You should now be able to

03

Proving it works

07–08

One command per stage. Debugging is walking them in order.

the chain, read off a live proxy · for whoever has to sign it off

  1. 07

    How do I read the chain off a live proxy?

    hands on

    Not knowing this costs

    Random poking finds the broken stage eventually. The walk finds it first.

    • One istioctl subcommand per stage
    • Walk them in chain order, stop at the first surprise
    shellthe four commands, in order
    istioctl proxy-config listeners deploy/checkout -n checkout
istioctl proxy-config routes    deploy/checkout -n checkout --name 8080
istioctl proxy-config cluster   deploy/checkout -n checkout \
  --fqdn payment-svc.payments.svc.cluster.local
istioctl proxy-config endpoint  deploy/checkout -n checkout \
  --cluster "outbound|8080||payment-svc.payments.svc.cluster.local"
    Door, decision, group, address. The first command whose output surprises you is the stage where your problem lives.
  2. 08

    What is on the admin interface?

    hands on

    Not knowing this costs

    Every istioctl answer is a summary. The dump is the source.

    • Port 15000, inside the pod, localhost only
    • The full config dump and every counter
    • The raw truth istioctl summarises
    shellthe unabridged version
    kubectl exec deploy/checkout -c istio-proxy -n checkout -- \
  pilot-agent request GET config_dump > /tmp/dump.json

kubectl exec deploy/checkout -c istio-proxy -n checkout -- \
  pilot-agent request GET stats | grep payment | head
    The config dump is enormous and complete: what this proxy believes, verbatim. When istioctl output and reality disagree, this is the tiebreaker.

Before you scroll on

0/3

You should now be able to

04

When it breaks

09–10

Every flag in an access log is one stage of the chain saying where it gave up.

the failures you will actually hit · for whoever gets paged

  1. 09

    How do response flags map to the chain?

    Not knowing this costs

    The flag tells you which command from rung 7 to run first.

    • NR: no route matched, stage two
    • UH and UO: cluster trouble, stage three
    • UF: the endpoint itself refused, stage four
  2. 10

    A route I wrote is missing from the proxy. Where did it go?

    hands on

    Not knowing this costs

    A stale proxy fails no health check while running week old routes.

    • istiod rejected or never sent the update
    • proxy-status shows STALE or the config NACKed
    • The proxy keeps serving the last good config
    shellfind the stuck proxy
    istioctl proxy-status | grep -v SYNCED

# NAME                     CDS     LDS     EDS     RDS
# checkout-x2k1.checkout   SYNCED  SYNCED  SYNCED  STALE  <- stuck on old routes
    STALE means istiod pushed and the proxy has not accepted. Last good config keeps serving, which is graceful and also why the symptom is your change silently absent.

Before you scroll on

0/3

You should now be able to

05

Where it ends

11–12

Envoy executes decisions. It does not make them.

the boundary of what the proxy is · for whoever reviews the design

  1. 11

    What does Envoy never decide?

    Not knowing this costs

    Debugging Envoy first inverts the order the bugs actually occur in.

    • Policy: it executes whatever config arrives
    • istiod decides, from your YAML
    • Blame order: your YAML, then istiod, then Envoy, almost always in that order
  2. 12

    What does this page unlock?

    Not knowing this costs

    Mesh debugging without this map is folklore. With it, it is a walk.

    • Every istioctl command now has a place on a map
    • Every response flag names a stage
    • Next: istiod, the thing writing all this config

Before you scroll on

0/3

You should now be able to

Go deeper

4 links, each earning its place.

Envoy life of a request↗

The chain from rung 4, told by the project itself, stage by stage.

Debugging Envoy and istiod↗

The four commands from rung 7, with their full option surface.

Envoy admin interface↗

Everything on port 15000 beyond the two endpoints rung 8 uses.

Envoy response flags↗

The full flag table that rung 9 compresses to three lines.

Where this leaves you

Rung 4 is the map. Rung 7 is the map as four commands you will run for the rest of your career.

If you keep one thing: listener, route, cluster, endpoint. Every proxy mystery is one of those four words wearing a costume.