Stop reading whenever you have enough

VirtualService, in eighteen questions.

Each one is the question the previous answer makes you ask. Twelve come with the manifest or the command, because that is the part you meet at work.

Not technical? The first three questions are written for you, and they are enough to follow any conversation about this.

Rungs
18
Hands on
12
Read
~11 min
01

What it is

01–03

Programs ask for a place by name. This is the instruction sheet deciding where that name actually leads.

three minutes, no cluster needed · for anyone

  1. 01

    What problem is this solving?

    Not knowing this costs

    Without it, changing where traffic goes means redeploying the callers.

    • A program asks for payments by name
    • Where that name leads used to be fixed
    • A VirtualService makes the destination a decision you write down
  2. 02

    What does that make possible?

    Not knowing this costs

    This is the mechanism behind every canary release and instant rollback.

    • Send one percent of users to a new version
    • Give up on a slow answer, and retry a failed one
    • All without changing a line of the calling program
  3. 03

    Is this not what a DestinationRule does?

    Not knowing this costs

    The most common 503 in Istio is these two disagreeing.

    • No. VirtualService decides where a request goes
    • DestinationRule decides what happens once it arrives
    • The groups one routes to, only the other can define

Before you scroll on

0/3

You should now be able to

02

The YAML you will see

04–07

Four small blocks: match this, send it there, wait this long, try again this many times.

the four blocks that appear in real repos · for whoever writes the manifests

  1. 04

    What does a canary route look like?

    hands on

    Not knowing this costs

    Routing to an undeclared subset 503s the exact slice you canaried.

    • A host, then routes evaluated top to bottom
    • Weights split traffic between named subsets
    • The subsets must exist in a DestinationRule first
    manifestone percent, for real users
    apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: payment-svc
  namespace: payments
spec:
  hosts:
    - payment-svc.payments.svc.cluster.local
  http:
    - route:
        - destination:
            host: payment-svc.payments.svc.cluster.local
            subset: v1
          weight: 99
        - destination:
            host: payment-svc.payments.svc.cluster.local
            subset: v2
          weight: 1
    Weights are per proxy, so tiny percentages are approximate at low traffic. And subset v2 must already exist in a DestinationRule, or this is an outage shaped like a canary.
  2. 05

    How do testers reach v2 before real users do?

    hands on

    Not knowing this costs

    Route order is silent. A shadowed route looks identical in git.

    • A match block on a header, above the weighted route
    • First matching route wins, evaluation stops there
    • The catch all must always be last
    manifestthe escape hatch, in order
      http:
    - match:                      # checked first, top to bottom
        - headers:
            x-release-channel:
              exact: internal
      route:
        - destination:
            host: payment-svc.payments.svc.cluster.local
            subset: v2
    - route:                      # the catch all, LAST
        - destination:
            host: payment-svc.payments.svc.cluster.local
            subset: v1
    Put the catch all first and the header route below it becomes unreachable. No error, no warning, testers silently land on v1 and sign off a version they never touched.
  3. 06

    How long should a request be allowed to take?

    hands on

    Not knowing this costs

    The invisible default retry doubles load on an upstream that is already failing.

    • timeout: the total ceiling for the route
    • Istio retries twice by default, even unconfigured
    • Write both down rather than inheriting them
    manifestthe ceiling, made explicit
      http:
    - route:
        - destination:
            host: payment-svc.payments.svc.cluster.local
            subset: v1
      timeout: 3s
      retries:
        attempts: 2
        perTryTimeout: 1s
        retryOn: connect-failure,refused-stream,unavailable
    The default is no timeout and two retries. So an unconfigured route already retries, which surprises everyone the first time a slow upstream gets hit three times.
  4. 07

    What must never be retried?

    hands on

    Not knowing this costs

    A default retry on a checkout route is a duplicate charge waiting for load.

    • Anything that changes something: payments, orders, sends
    • Give those routes attempts zero, explicitly
    • retryOn connect-failure alone is not safe enough
    manifestthe non idempotent route, fenced off
      http:
    - match:
        - uri:
            prefix: /api/checkout/submit
      route:
        - destination:
            host: payment-svc.payments.svc.cluster.local
            subset: v1
      timeout: 10s
      retries:
        attempts: 0        # a duplicate here is a duplicate charge
    A retry can fire after the upstream did the work but before the response arrived. For a submit endpoint that is the same purchase twice, working exactly as configured.

Before you scroll on

0/3

You should now be able to

03

Proving it works

08–09

Trust nothing you cannot test. Two commands show the routes the proxy actually holds.

commands that answer yes or no · for whoever has to sign it off

  1. 08

    What routes does the proxy actually hold?

    hands on

    Not knowing this costs

    kubectl apply proves the YAML parsed. Only the proxy knows what it runs.

    • Ask a caller side proxy, not the YAML
    • Weights and match rules read back exactly as programmed
    shellthe routes, from the horse
    istioctl proxy-config routes deploy/checkout -n checkout \
  --name 8080 -o json | grep -E 'weight|prefix|exact' -A1

# "exact": "internal"      <- the header route exists
# "weight": 99
# "weight": 1              <- the split the proxy will actually apply
    Config distributes in seconds normally, but a NACKed config stays stale silently. This read is the difference between applied and in force.
  2. 09

    How do I prove the split with curl?

    hands on

    Not knowing this costs

    A canary you never observed is a belief, not a rollout.

    • Every Istio response carries the server header
    • Ask the version endpoint many times, count the answers
    • Send the tester header once, expect v2 every time
    shellcount, then force
    kubectl exec deploy/sleep -n payments -- sh -c \
  'for i in $(seq 1 100); do curl -s payment-svc:8080/version; done | sort | uniq -c'
#   99 v1
#    1 v2        <- the weighted split, observed

kubectl exec deploy/sleep -n payments -- \
  curl -s -H "x-release-channel: internal" payment-svc:8080/version
# v2            <- the escape hatch, every single time
    The header test doubles as the route order test: if it ever returns v1, the catch all is above the match block and rung 5 is your bug.

Before you scroll on

0/3

You should now be able to

04

When it breaks

10–12

The ways this fails all look mysterious from the outside. None of them are.

the failures you will actually hit · for whoever gets paged

  1. 10

    The canary slice 503s instantly. What is it?

    hands on

    Not knowing this costs

    Both files pass review alone. Only the pair is broken.

    • Flag UH, near 0ms, empty upstream_host
    • The route points at a subset no rule declares
    • Fix the DestinationRule, not the pods
    shellthirty seconds to certainty
    kubectl logs deploy/checkout -c istio-proxy -n checkout --tail=5
# ... 503 UH "-" ... duration=0 upstream_host="-"

istioctl proxy-config cluster deploy/checkout -n checkout \
  --fqdn payment-svc.payments.svc.cluster.local
# no v2 row -> the subset was never programmed
    Istio validates the two resources separately and never checks that a subset named in one exists in the other. istioctl analyze in CI catches it before traffic does.
  2. 11

    My new match rule does nothing. Why?

    hands on

    Not knowing this costs

    A shadowed route fails no test. It just never runs.

    • A route above it already matches everything
    • First match wins, evaluation never reaches yours
    • Read the routes in proxy order, not file order
    shellfind the shadow
    istioctl proxy-config routes deploy/checkout -n checkout --name 8080

# NAME     DOMAINS          MATCH        VIRTUAL SERVICE
# 8080     payment-svc      /*           payment-svc.payments   <- catch all FIRST
# 8080     payment-svc      /api/beta*   payment-svc.payments   <- unreachable
    Also true across files: two VirtualServices on one host merge in an order you do not control. Keep one VirtualService per host, same as the DestinationRule ownership rule.
  3. 12

    Latency doubled but nothing errors. What is it?

    hands on

    Not knowing this costs

    Retries hide an upstream failure until the day they amplify it.

    • Retries: failures being converted into delay
    • The upstream sees more requests than callers sent
    • Response flag URX marks a retry exhausted request
    shellsee the amplification
    kubectl exec deploy/checkout -c istio-proxy -n checkout -- \
  pilot-agent request GET stats | grep retry

# ...upstream_rq_retry: 4183            <- retries happening now
# ...upstream_rq_retry_success: 3900    <- mostly rescuing failures
# ...upstream_rq_retry_overflow: 0      <- budget not yet clipping
    retry_success high means retries are masking a real upstream problem. You are not down, but only because every failure is being paid for twice in latency.

Before you scroll on

0/3

You should now be able to

05

Beyond the basics

13–15

Once routing works, you can copy real traffic harmlessly and rehearse failures on purpose.

the parts that appear at fleet scale · for whoever owns the platform

  1. 13

    Can I test v2 with real traffic and zero risk?

    hands on

    Not knowing this costs

    Mirroring a service that sends emails sends every email twice.

    • Mirror a percentage to v2, responses discarded
    • Users are served by v1 exactly as before
    • v2 sees genuine production requests, harmlessly
    manifestthe shadow copy
      http:
    - route:
        - destination:
            host: payment-svc.payments.svc.cluster.local
            subset: v1
      mirror:
        host: payment-svc.payments.svc.cluster.local
        subset: v2
      mirrorPercentage:
        value: 10.0
    Mirrored requests carry a -shadow suffix on the Host header. Make sure v2 side effects are safe: a mirror that writes to the production database is not harmless.
  2. 14

    Can I rehearse a failure before it happens?

    hands on

    Not knowing this costs

    The first test of your timeout should not be a real incident.

    • Fault injection: manufactured delays and errors
    • Gate it behind a header so only the drill sees it
    • Prove the timeout works before the outage does
    manifestchaos, for one person only
      http:
    - match:
        - headers:
            x-chaos-drill:
              exact: latency-2s
      fault:
        delay:
          fixedDelay: 2s
          percentage:
            value: 100
      route:
        - destination:
            host: payment-svc.payments.svc.cluster.local
            subset: v1
    An injected delay above the caller timeout from rung 6 proves the whole failure path: timeout fires, retry fires, fallback renders. Without the header gate this is an outage you wrote yourself.
  3. 15

    What stops retries stampeding the whole fleet?

    hands on

    Not knowing this costs

    A retry storm turns one slow service into a mesh wide brownout.

    • Every caller retrying twice can triple total load
    • Envoy caps concurrent retries with a shared budget
    • Watch the overflow counter, not the retry counter
    shellthe stampede, measured
    kubectl exec deploy/checkout -c istio-proxy -n checkout -- \
  pilot-agent request GET stats | grep retry_overflow

# ...upstream_rq_retry_overflow: 812   <- budget clipping retries NOW
    Overflow rising during an incident is the budget doing its job: shedding retries so the upstream can recover. Raising retries in response is the exact wrong move.

Before you scroll on

0/3

You should now be able to

06

Where it ends

16–18

What this file does not decide, which is the part that gets misassigned in reviews.

the limits, where people get caught · for whoever reviews the design

  1. 16

    What does a VirtualService not define?

    Not knowing this costs

    Rung 10 is this misunderstanding, deployed.

    • Not subsets: those live in the DestinationRule
    • Not connection limits or ejection, same file
    • Naming a group is not creating one
  2. 17

    Is routing a security boundary?

    Not knowing this costs

    A route hiding an endpoint is not the same as protecting it.

    • No. Rules live in proxies, and only in proxies
    • Anything reaching a pod IP directly skips them all
    • Allowed or not is AuthorizationPolicy
  3. 18

    What did we trade away for all this?

    Not knowing this costs

    Debugging without knowing a VirtualService exists costs teams days.

    • Where a request goes is no longer in the code
    • A developer reading the program cannot see the one percent
    • The routing file is now production critical config

Before you scroll on

0/3

You should now be able to

Go deeper

5 links, each earning its place.

VirtualService reference↗

The authoritative field list, including the match, retry and mirror fields this page compresses.

Traffic shifting task↗

The official runnable canary: rungs 4 and 9 on a live cluster, step by step.

Fault injection task↗

The drill from rung 14 in full, including verifying the delay actually reached the user.

Mirroring task↗

The shadow traffic setup from rung 13, with the logs that prove the mirror is flowing.

Envoy response flags↗

The definitive table for UH, URX and friends. It turns a generic 503 into a narrow hypothesis in seconds.

Where this leaves you

Rungs 1 to 3 are what most people need. Rungs 8 and 9 are the two commands that turn a canary from a belief into an observation.

If you keep one thing: routes are evaluated top to bottom and the first match wins. Half of everything that goes wrong here is that sentence, ignored.