Stop reading whenever you have enough

Traffic mirroring, in twelve questions.

Each one is the question the previous answer makes you ask. The ones that matter come with the manifest or the command, because that is the part you meet at work.

Not technical? The first three questions are written for you, and they are enough to follow any conversation about this.

Rungs
12
Hands on
4
Read
~8 min
01

What it is

01–03

Every real request is photocopied to the new version. The copies answers go in the bin, so nobody is affected.

three minutes, no cluster needed · for anyone

  1. 01

    What problem is this solving?

    Not knowing this costs

    It is the only test that is both real and free of user risk.

    • Staging traffic is fake and finds fake bugs
    • Canaries expose real users to find real ones
    • Mirroring gets real traffic with zero user exposure
  2. 02

    How is this different from a canary?

    Not knowing this costs

    Mirroring answers does it crash. Only a canary answers is it better.

    • A canary serves real users from the new version
    • A mirror never serves anyone from it
    • Mirror first, canary second is the natural order
  3. 03

    Why can the copy not hurt the user?

    Not knowing this costs

    This guarantee covers responses only. Side effects are rung 9.

    • The proxy sends it fire and forget
    • The response is discarded unread
    • A crashed shadow changes nothing for the caller

Before you scroll on

0/3

You should now be able to

02

The YAML you will see

04–05

A destination for the copies and a percentage. That is the whole surface.

the mirror block and its two dials · for whoever writes the manifests

  1. 04

    What does the mirror block look like?

    hands on

    Not knowing this costs

    The default is one hundred percent, and defaults ship.

    • A mirror destination beside the real route
    • mirrorPercentage dials the copy volume
    • The shadow subset must exist in a DestinationRule
    manifestten percent, photocopied
      http:
    - route:
        - destination:
            host: payment-svc.payments.svc.cluster.local
            subset: stable
      mirror:
        host: payment-svc.payments.svc.cluster.local
        subset: v2
      mirrorPercentage:
        value: 10.0
    Omit mirrorPercentage and everything is mirrored, which doubles the shadow load instantly. Start at ten and turn it up on purpose, not by default.
  2. 05

    How does the shadow know a request is a copy?

    Not knowing this costs

    A shadow that cannot tell it is a shadow will act like it is real.

    • The Host header arrives with a -shadow suffix
    • That is the only built in marker
    • Shadow aware code can branch on it if it must

Before you scroll on

0/3

You should now be able to

03

Proving it works

06–08

The mirror is invisible to users, so you prove it from the shadow side.

commands that answer yes or no · for whoever has to sign it off

  1. 06

    Are copies actually arriving?

    hands on

    Not knowing this costs

    A silent mirror produces confident conclusions from zero data.

    • Watch the shadow logs for the suffixed host
    • Volume should track the percentage you set
    shellthe shadow side, observed
    kubectl logs deploy/payment-svc-v2 -c istio-proxy -n payments --tail=20 \
  | grep shadow

# ... "GET /api/charge" 200 ... payment-svc-shadow ...

# rate check: shadow requests / live requests ~= 0.10
sum(rate(istio_requests_total{destination_version="v2"}[5m]))
/ sum(rate(istio_requests_total{destination_version="v1"}[5m]))
    The ratio drifting far from your percentage usually means another route is bypassing the mirror block, not that mirroring is broken.
  2. 07

    How do I judge the shadow, with no users on it?

    hands on

    Not knowing this costs

    This comparison is the entire payoff of running the mirror.

    • Compare its error rate and latency against live
    • Same requests, so the comparison is honest
    • Diff logs for divergent status codes per path
    shelllive and shadow, side by side
    sum by (destination_version, response_code) (
  rate(istio_requests_total{
    destination_service_name="payment-svc"}[10m]))

# v1  200: 94.1   v2  200: 9.38    <- tracking at 10%
# v1  500: 0.02   v2  500: 1.90    <- shadow failing requests live serves
    The shadow failing on requests live serves fine is exactly the bug class mirroring exists to catch, found before any user met it.
  3. 08

    Does a dying shadow really cost users nothing?

    hands on

    Not knowing this costs

    Guarantees you never tested are guarantees you are borrowing.

    • Kill the shadow during the test window
    • Live latency and errors must not move
    shellprove the guarantee, once
    kubectl scale deploy/payment-svc-v2 -n payments --replicas=0

# then watch the live series stay flat:
histogram_quantile(0.99, sum by (le) (
  rate(istio_request_duration_milliseconds_bucket{
    destination_version="v1"}[5m])))
    Fire and forget is the documented behaviour; this makes it your measured behaviour. Ten minutes, once, and the guarantee is yours instead of the docs.

Before you scroll on

0/3

You should now be able to

04

When it breaks

09–10

Harmless is a property of the shadow service, not of mirroring. That is where it bites.

the failures you will actually hit · for whoever gets paged

  1. 09

    What makes a mirror genuinely dangerous?

    Not knowing this costs

    Mirroring a service that sends email is a duplicate email campaign.

    • The shadow does real side effects with copied requests
    • Emails send twice, webhooks fire twice, rows write twice
    • Point the shadow at stubs or a scratch datastore
  2. 10

    Where does the extra load land?

    Not knowing this costs

    A mirror at one hundred percent doubles read load on shared stores.

    • The caller proxy sends every request twice
    • Shared dependencies of the shadow feel real load
    • The database serves both versions of every read

Before you scroll on

0/3

You should now be able to

05

Where it ends

11–12

A perfect mirror still cannot tell you everything a canary can.

the limits, where people get caught · for whoever reviews the plan

  1. 11

    What can a mirror not tell you?

    Not knowing this costs

    A mirror validates the request path. The response path ships untested.

    • Whether users like the new behaviour
    • Anything about responses, nobody reads them
    • Write correctness, if writes were stubbed out
  2. 12

    When does the mirror come down?

    Not knowing this costs

    Every long lived mirror eventually pages someone as a mystery.

    • It is scaffolding, not architecture
    • When the comparison is clean, promote to a canary
    • A forgotten mirror is permanent double load

Before you scroll on

0/3

You should now be able to

Go deeper

3 links, each earning its place.

Mirroring task↗

The official runnable setup, including the log check from rung 6.

HTTPRoute mirror reference↗

The mirror and mirrorPercentage fields, including the hundred percent default from rung 4.

Traffic shifting task↗

The canary that should follow a clean mirror, which is the natural next page.

Where this leaves you

Rung 7 is the payoff: the shadow failing requests live serves fine, found before any user met it.

If you keep one thing: mirroring makes responses harmless, not side effects. Harmless is a property you build into the shadow, not one the mesh grants it.