Stop reading whenever you have enough

Canary releases, in twelve questions.

Each one is the question the previous answer makes you ask. The ones that matter come with the manifest or the command, because that is the part you meet at work.

Not technical? The first three questions are written for you, and they are enough to follow any conversation about this.

Rungs
12
Hands on
7
Read
~8 min
01

What it is

01–03

Instead of giving everyone the new version at once, you give it to a few and watch.

three minutes, no cluster needed · for anyone

  1. 01

    What problem is this solving?

    Not knowing this costs

    The blast radius of a bad deploy is a choice, not a fact.

    • Every deploy is a bet that the new version works
    • All at once bets every user on it
    • A canary bets one percent, then watches
  2. 02

    Why does one percent tell you anything?

    Not knowing this costs

    The bugs that page you are the ones staging could not produce.

    • Real traffic finds what staging never does
    • Real payloads, real devices, real timing
    • One percent of real beats one hundred percent of fake
  3. 03

    Who decides the canary is good?

    Not knowing this costs

    A canary without a comparison is just a slow deploy.

    • Numbers do: error rate and latency, canary versus stable
    • Compared over a fixed window, before each stage
    • A human or a tool applies the same rule either way

Before you scroll on

0/3

You should now be able to

02

The rollout you will run

04–06

A canary is a sequence of weights with a decision between each one.

the stages and the manifests behind them · for whoever ships the release

  1. 04

    What do the two resources look like?

    hands on

    Not knowing this costs

    Order of apply is the difference between a canary and an outage.

    • DestinationRule names the versions as subsets
    • VirtualService splits the traffic by weight
    • The subset must exist before any route names it
    manifestthe pair, reviewed together
    # DestinationRule: the groups exist
  subsets:
    - name: stable
      labels: { version: v1 }
    - name: canary
      labels: { version: v2 }
---
# VirtualService: the split
  http:
    - route:
        - destination:
            host: payment-svc.payments.svc.cluster.local
            subset: stable
          weight: 99
        - destination:
            host: payment-svc.payments.svc.cluster.local
            subset: canary
          weight: 1
    Apply the DestinationRule first, confirm the subset cluster exists, then apply the split. The reverse order 503s the canary slice instantly.
  2. 05

    What are the stages?

    hands on

    Not knowing this costs

    Stages without soak time are one big bang wearing four costumes.

    • One percent, ten, fifty, one hundred
    • A fixed soak window at each stage
    • The same metric gate between every pair
    shelleach stage is one edit
    kubectl -n payments patch virtualservice payment-svc --type=json -p='[
  {"op":"replace","path":"/spec/http/0/route/0/weight","value":90},
  {"op":"replace","path":"/spec/http/0/route/1/weight","value":10}
]'

# soak, compare, then 50/50, then 0/100
    Weights must sum to one hundred or the VirtualService is rejected. In practice this patch lives in a pipeline, not in a terminal, but it is this patch.
  3. 06

    How do I roll back?

    hands on

    Not knowing this costs

    A rollback that needs a build takes an hour. This takes seconds.

    • Set the canary weight to zero
    • Seconds to take effect, no pods restarted
    • The broken version keeps running, receiving nothing
    shellthe whole rollback
    kubectl -n payments patch virtualservice payment-svc --type=json -p='[
  {"op":"replace","path":"/spec/http/0/route/0/weight","value":100},
  {"op":"replace","path":"/spec/http/0/route/1/weight","value":0}
]'
# done. the canary pods still run, useful for reading logs
    This is why canaries beat rolling restarts for risk: rollback is a config change, not a redeploy, and the evidence stays up for the postmortem.

Before you scroll on

0/3

You should now be able to

03

Proving it works

07–08

Between stages you compare the canary against the old version with numbers, not feelings.

the gate that decides, as a query · for whoever approves the next stage

  1. 07

    Is the canary actually failing more than stable?

    hands on

    Not knowing this costs

    Promoting on gut feel is how a one percent incident becomes a hundred.

    • Compare error rates by destination version
    • Same service, same window, two lines
    shellthe gate, as a query
    # error rate per version, last 10 minutes
sum by (destination_version) (
  rate(istio_requests_total{destination_service_name="payment-svc",
    response_code=~"5.."}[10m]))
/
sum by (destination_version) (
  rate(istio_requests_total{destination_service_name="payment-svc"}[10m]))

# {destination_version="v1"} 0.001
# {destination_version="v2"} 0.004   <- 4x stable: do not promote
    Compare against stable in the same window, never against an absolute threshold. If stable is also failing, the problem is not your canary.
  2. 08

    Is the canary slower, even without errors?

    hands on

    Not knowing this costs

    Error rate clean and latency doubled still ruins one user in a hundred.

    • Compare P99 by version, not the average
    • Regressions hide in the tail first
    shelllatency, version against version
    histogram_quantile(0.99, sum by (destination_version, le) (
  rate(istio_request_duration_milliseconds_bucket{
    destination_service_name="payment-svc"}[10m])))

# {destination_version="v1"} 180
# {destination_version="v2"} 460   <- tail doubled: hold the stage
    P50 identical and P99 doubled is a real regression, usually a cold cache or a new N+1 query. It will not improve by promoting.

Before you scroll on

0/3

You should now be able to

04

When it breaks

09–10

Canaries fail in two ways: the new version is bad, or the canary itself was wired wrong.

the failures you will actually hit · for whoever gets paged mid rollout

  1. 09

    The canary slice 503s the moment weights shift. What is it?

    hands on

    Not knowing this costs

    Rolling back pods cannot fix a failure that lives in two YAML files.

    • Instant 503 at zero milliseconds, so nothing was ever dialled
    • Flag NR: no cluster exists for the subset you named
    • Weights back to zero, fix the pair, start again
    shellconfirm before blaming v2
    istioctl proxy-config cluster deploy/checkout -n checkout \
  --fqdn payment-svc.payments.svc.cluster.local
# no canary row -> the rule never landed. v2 is innocent
    Read the flag carefully: NR means the subset was never programmed, so the DestinationRule is missing or landed late. UH means the cluster exists but holds no healthy pod, which is a labels or readiness problem instead. Different flag, different file. istioctl analyze in the pipeline catches the NR case before any weight moves.
  2. 10

    Why do some users flap between versions?

    hands on

    Not knowing this costs

    A user seeing two UIs in one session files the bug you cannot reproduce.

    • Weights are per request, not per user
    • One user can hit v1, then v2, then v1
    • Consistent hashing pins a user to one version
    manifeststickiness, when the UI needs it
    # DestinationRule, on the same host
  trafficPolicy:
    loadBalancer:
      consistentHash:
        httpCookie:
          name: canary-affinity
          ttl: 3600s
    Only needed when the two versions render differently enough that flapping is visible. For a pure API change, per request weights are fine and simpler.

Before you scroll on

0/3

You should now be able to

05

Where it ends

11–12

Some changes cannot be canaried, and knowing which is the senior skill.

the limits, where people get caught · for whoever reviews the release plan

  1. 11

    What can a canary not make safe?

    Not knowing this costs

    The worst canary incidents are stable users hurt by the canary sideways.

    • Anything both versions share: the database, the queue
    • A bad migration hurts stable users through the canary
    • Schema changes need expand and contract, not weights
  2. 12

    What does the one percent stage actually prove?

    Not knowing this costs

    A memory leak that takes six hours will pass a ten minute gate.

    • Crashes, error spikes, gross latency regressions
    • Not rare bugs: one percent of traffic finds one percent bugs slowly
    • Soak time, not stage count, catches the slow leak

Before you scroll on

0/3

You should now be able to

Go deeper

4 links, each earning its place.

Traffic shifting task↗

The official runnable version of rungs 4 and 5 on a live cluster.

Flagger, progressive delivery operator↗

The stages and gates from this page, automated: it edits the same weights and runs the same queries.

Istio standard metrics↗

Every label the gate queries in rungs 7 and 8 can use, defined.

Argo Rollouts↗

The other mainstream automation of this exact process, if your platform is Argo shaped.

Where this leaves you

Rungs 4 to 6 are the mechanics. Rungs 7 and 8 are the two queries that make it a canary instead of a slow deploy.

If you keep one thing: rollback is editing one number to zero. Anything that makes rollback harder than that is removing the reason canaries exist.