Stop reading whenever you have enough

Production Istio, in thirteen questions.

Each one is the question the previous answer makes you ask. The ones that matter come with the command, because that is the part you meet at work.

Not technical? The first two questions are written for you, and they are enough to follow any conversation about this.

Rungs
13
Hands on
4
Read
~9 min
01

What production means here

01–02

Installing the mesh is an afternoon. Operating it is upgrades, alerts and habits, forever.

the difference between installed and operated · for anyone

  1. 01

    What changes when real traffic depends on the mesh?

    Not knowing this costs

    A mesh nobody owns operationally is an outage on layaway.

    • Every mesh bug is now a customer bug
    • Every upgrade is a bet with revenue on the table
    • The mesh becomes a product with users: your teams
  2. 02

    What are the three ongoing duties?

    Not knowing this costs

    Teams that only installed discover the duties during incidents.

    • Upgrading: istiod, proxies, and their skew
    • Watching: the five alerts, honestly wired
    • Gatekeeping: config quality at the door, in CI

Before you scroll on

0/3

You should now be able to

02

Upgrades without drama

03–05

Two control planes run side by side, and namespaces walk from old to new at your pace.

revisions, canaried control planes, restart debt · for whoever runs the upgrade

  1. 03

    Why are revisions the whole upgrade story?

    Not knowing this costs

    In place upgrades bet the whole mesh on one apply.

    • A revision is a second istiod, side by side
    • Namespaces choose their control plane by label
    • Rollback is relabelling, not reinstalling
  2. 04

    What does the canary upgrade actually look like?

    hands on

    Not knowing this costs

    One namespace at a time converts upgrade risk into a walk.

    • Install the new revision, move one quiet namespace
    • Restart its workloads, verify, then proceed
    • The old revision keeps serving everyone else
    shelltwo brains, one migration
    istioctl install --set revision=1-24-2 -y

kubectl label namespace payments istio.io/rev=1-24-2 istio-injection- --overwrite
kubectl rollout restart deploy -n payments

istioctl proxy-status | grep payments
# ...payments  SYNCED ... istiod-1-24-2   <- on the new brain, verified
    The injection label and the rev label are mutually exclusive: the old one must come off. Leaving both is the classic why is nothing moving of upgrade day.
  3. 05

    When is the upgrade actually done?

    hands on

    Not knowing this costs

    Two permanent revisions is not safety, it is two upgrade debts.

    • When no proxy reports the old version
    • Then the old revision uninstalls
    • Restart debt is the part everyone defers
    shellthe done check, and the cleanup
    istioctl proxy-status | awk '{print $NF}' | sort | uniq -c
#  450 istiod-1-24-2    <- everyone moved
#    0 istiod-1-23-1

istioctl uninstall --revision=1-23-1 -y
    Uninstalling the old revision while stragglers still point at it orphans them mid flight. The count reaching zero is the gate, not the calendar.

Before you scroll on

0/3

You should now be able to

03

The alerts that matter

06–07

Five alerts cover the mesh. More than that is noise wearing a pager.

the short list that catches real incidents · for whoever writes the alert rules

  1. 06

    What are the five alerts?

    Not knowing this costs

    These five catch config, identity, distribution, edge and capacity.

    • istiod down, and root or intermediate cert expiry
    • Proxies out of sync, fleet wide
    • Gateway 5xx rate, and mesh wide UH or UO spikes
  2. 07

    What do the two quietest alerts look like as queries?

    hands on

    Not knowing this costs

    Every mesh that paged on cert expiry had monitoring, just not these.

    • Cert expiry from the istiod gauge
    • Config distribution slowing, or being rejected
    shellthe two nobody writes until after
    # root cert expiry, alert at 90 days out:
(citadel_server_root_cert_expiry_timestamp - time()) / 86400 < 90

# config taking longer to reach proxies, alert for 15m:
histogram_quantile(0.99, sum by (le) (
  rate(pilot_proxy_convergence_time_bucket[5m]))) > 5

# istiod refusing config outright:
rate(pilot_total_xds_rejects[5m]) > 0
    No metric names the individual stale pod, so pair these with the proxy-status check as a scheduled job: the metrics tell you distribution is unhealthy, proxy-status tells you which pods. Both alert on silence, which is why they get written after the first incident instead of before it.

Before you scroll on

0/3

You should now be able to

04

The habits

08–10

GitOps for the YAML, gates in CI, and a version skew you never let widen.

what disciplined mesh teams do weekly · for whoever owns the platform

  1. 08

    What gates a mesh change before it merges?

    hands on

    Not knowing this costs

    Every classic from the troubleshooting ladder is cheaper here.

    • analyze against the live cluster, in CI
    • A failed check blocks the merge, no exceptions
    • The dangling subset dies in review, not at 3am
    shellthe CI gate, whole
    istioctl analyze -A --use-kube \
  --failure-threshold=Warning && echo GATE-PASS

# exits non zero on findings: wire it as a required check
    Analyze in CI catches cross resource breakage that file linting cannot see: the subset in one PR that a VirtualService from last month depends on.
  2. 09

    How much version skew is allowed?

    Not knowing this costs

    Skew never breaks on the day it widens. It breaks on upgrade day.

    • Data plane at most one minor behind istiod
    • Wider skew is unsupported, quietly
    • The straggler count is a weekly chore, not a report
  3. 10

    What belongs in git, and what never does?

    Not knowing this costs

    The unreviewed patch during an incident becomes permanent by forgetting.

    • Every mesh resource, applied by GitOps only
    • Never: ad hoc kubectl on production mesh config
    • The cluster is a projection of the repo, or chaos

Before you scroll on

0/3

You should now be able to

05

Where it ends

11–13

Some risk does not tune away. Naming it is part of operating honestly.

the limits, where people get caught · for whoever reviews the platform

  1. 11

    What single points remain, whatever you do?

    Not knowing this costs

    Pretending these away is worse than naming and watching them.

    • istiod and its certificate chain
    • The registry serving proxy images at pod start
    • Whoever holds admin on istio-system
  2. 12

    What does the mesh team owe the app teams?

    Not knowing this costs

    A platform without a paved road becomes twenty bespoke meshes.

    • A paved road: templates that pass the gates
    • The five alerts on their behalf, not per team
    • An upgrade calendar they hear about before it lands
  3. 13

    What gets revisited yearly, on purpose?

    Not knowing this costs

    Platforms that never re justify themselves get cancelled by surprise.

    • Whether ambient or sidecarless changes the bill
    • Whether the skew, alerts and gates still hold
    • Whether the mesh still earns its operational cost

Before you scroll on

0/3

You should now be able to

Go deeper

4 links, each earning its place.

Canary upgrades with revisions↗

The official version of rungs 3 to 5, including the label mechanics.

Istio supported releases and skew↗

The exact support windows behind rung 9.

istioctl analyze↗

The CI gate from rung 8, with every flag.

Istio operations best practices↗

The project’s own habits list, overlapping and extending this band.

Where this leaves you

Rungs 4 and 5 are the upgrade, which is the duty that frightens people. Rung 7 holds the two alerts that prevent the loudest pages.

If you keep one thing: revisions make upgrades a walk between two running control planes. Everything scary about operating Istio shrinks once that is routine.