Stop reading whenever you have enough

DestinationRule, in eighteen questions.

Each one is the question the previous answer makes you ask. Fourteen come with the manifest or the command, because that is the part you meet at work.

Not technical? The first three questions are written for you, and they are enough to follow any conversation about this.

Rungs
18
Hands on
14
Read
~11 min
01

What it is

01–03

One name stands for many identical copies of a program. This is the written rulebook for dealing with the copies.

three minutes, no cluster needed · for anyone

  1. 01

    What problem is this solving?

    Not knowing this costs

    Without one, defaults decide, and defaults ignore a failing copy.

    • One service name stands for many identical copies
    • Something must pick a copy, limit it, and notice failures
    • A DestinationRule is those decisions, written down
  2. 02

    What exactly does it decide?

    Not knowing this costs

    Three decisions. Anything else you have heard belongs to a different resource.

    • Which copy gets each request
    • How many requests may be in flight at once
    • When to stop using a copy that keeps failing
  3. 03

    Is this not what a VirtualService does?

    Not knowing this costs

    The most common 503 in Istio is these two disagreeing.

    • No. VirtualService decides where a request goes
    • DestinationRule decides what happens once it arrives
    • They are written and reviewed as a pair

Before you scroll on

0/3

You should now be able to

02

The YAML you will see

04–08

Five small blocks of text: who you mean, the groups, the choosing, the limits, and the benching.

the five blocks that appear in real repos · for whoever writes the manifests

  1. 04

    What does the smallest useful rule look like?

    hands on

    Not knowing this costs

    A short host name is a rule that does nothing, invisibly.

    • A host, and one policy under trafficPolicy
    • The host must be the full name
    • Short names resolve against the rule namespace, not yours
    manifestthe host is the whole game
    apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: payment-svc
  namespace: payments
spec:
  host: payment-svc.payments.svc.cluster.local  # always fully qualified
  trafficPolicy:
    loadBalancer:
      simple: LEAST_REQUEST
    host: payment-svc written from another namespace binds to a service that does not exist, and the rule silently applies to nothing. Nothing errors.
  2. 05

    How do I split copies into versions?

    hands on

    Not knowing this costs

    A dangling subset is an instant 503 for exactly the canary slice.

    • Subsets: named groups selected by pod labels
    • The name is what VirtualServices route to
    • Only a DestinationRule can create one
    manifestthe groups a canary needs
    spec:
  host: payment-svc.payments.svc.cluster.local
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2          # must exist BEFORE any route names it
      labels:
        version: v2
    A VirtualService may route to a subset no rule declares. Istio accepts both files, programs a route to nothing, and every matching request fails the moment traffic shifts.
  3. 06

    Which copy should get each request?

    hands on

    Not knowing this costs

    Under ROUND_ROBIN a cold JVM pod takes full traffic on second one.

    • LEAST_REQUEST for most HTTP services
    • It routes around a slow copy; ROUND_ROBIN never does
    • Add warmup so cold starts ramp instead of drowning
    manifestthe sensible default, stated
    spec:
  host: payment-svc.payments.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      simple: LEAST_REQUEST
      warmupDurationSecs: 60s   # ramp new pods in, not straight to full share
    ROUND_ROBIN counts turns, not queue depth, so a slow pod keeps its full share. LEAST_REQUEST samples two and picks the emptier, which self corrects.
  4. 07

    How do I stop a slow dependency drowning me?

    hands on

    Not knowing this costs

    An unbounded queue turns a visible failure into silent slowness.

    • Cap connections, and cap the queue behind them
    • Keep the queue a small multiple of the connections
    • Past the caps, fail fast instead of waiting
    manifestboth numbers matter
    spec:
  host: payment-svc.payments.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 32   # the queue, kept short on purpose
    Raising the pending queue to make 503s disappear does not fix overload. It hides it as tail latency: P99 doubles while P50 and the error rate never move.
  5. 08

    How does a failing copy get benched?

    hands on

    Not knowing this costs

    maxEjectionPercent 100 converts a partial degradation into a total outage.

    • Outlier detection: consecutive errors eject an endpoint
    • It returns after baseEjectionTime and must reoffend
    • maxEjectionPercent keeps a floor of survivors
    manifestbenching, with a floor
    spec:
  host: payment-svc.payments.svc.cluster.local
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 34    # never eject the whole panel
    People read 34 and assume 100 is safer. Correlated failure is the common case, and 100 lets one shared bad dependency empty the entire cluster at once.

Before you scroll on

0/3

You should now be able to

03

Proving it works

09–11

Applying a file proves nothing. Three commands show what the proxies were actually told.

commands that answer yes or no · for whoever has to sign it off

  1. 09

    Did my subsets actually get programmed?

    hands on

    Not knowing this costs

    A route to a missing cluster is accepted silently and fails loudly.

    • Ask a caller proxy which clusters it holds
    • One cluster per subset, or the subset does not exist
    shellbefore any traffic shifts
    istioctl proxy-config cluster deploy/checkout -n checkout \
  --fqdn payment-svc.payments.svc.cluster.local

# SERVICE FQDN                               PORT  SUBSET
# payment-svc.payments.svc.cluster.local     8080  -
# payment-svc.payments.svc.cluster.local     8080  v1
# payment-svc.payments.svc.cluster.local     8080  v2   <- must be here
    No v2 row means the route will land on a cluster that was never built. This check belongs in the pipeline, before the traffic shift, not after the page.
  2. 10

    Does the subset have anything behind it?

    hands on

    Not knowing this costs

    Running, Ready pods prove nothing about whether Envoy can see them.

    • A subset cluster can exist with zero endpoints
    • Labels selecting nothing is the usual cause
    • Compare against real pod labels, not deployment labels
    shellthe empty subset check
    istioctl proxy-config endpoint deploy/checkout -n checkout \
  --cluster "outbound|8080|v2|payment-svc.payments.svc.cluster.local"

# ENDPOINT            STATUS      CLUSTER
# 10.4.1.9:8080       HEALTHY     outbound|8080|v2|...   <- good
# (header only, no rows)                                 <- labels match nothing
    version: v2 in the rule against app.kubernetes.io/version: v2 on the pods matches nothing. So does a label on the Deployment instead of the pod template.
  3. 11

    Are my limits actually in force?

    hands on

    Not knowing this costs

    kubectl apply succeeding tells you the YAML parsed, nothing more.

    • Read the effective cluster config, not your manifest
    • Envoy defaults showing means the rule bound to nothing
    shellthe tell is one absurd number
    istioctl proxy-config cluster deploy/checkout -n checkout \
  --fqdn payment-svc.payments.svc.cluster.local -o json \
  | grep -A 4 maxConnections

# "maxConnections": 100          <- your rule, in force
# "maxConnections": 4294967295   <- Envoy default: rule applied to nothing
    4294967295 is the unset default. Seeing it after you applied a cap means the host did not match, and the short name from rung 4 is the first suspect.

Before you scroll on

0/3

You should now be able to

04

When it breaks

12–14

The ways this fails all look mysterious from the outside. None of them are.

the three failures you will actually hit · for whoever gets paged

  1. 12

    The canary went out and instantly 503s. What is it?

    hands on

    Not knowing this costs

    Rolling back the pods cannot fix a failure the pods never saw.

    • Flag UH, near 0ms, empty upstream_host
    • The request never left the caller sidecar
    • Route names a subset, no rule declares it
    shellthirty seconds to certainty
    kubectl logs deploy/checkout -c istio-proxy -n checkout --tail=5
# ... 503 UH "-" ... duration=0 upstream_host="-"

istioctl proxy-config cluster deploy/checkout -n checkout \
  --fqdn payment-svc.payments.svc.cluster.local
# no v2 row  ->  dangling subset, fix the DestinationRule
    UH at zero milliseconds means no endpoint even existed to try. The upstream pods are irrelevant, however unhealthy the dashboard claims they are.
  2. 13

    Traffic fails in waves, recovering every thirty seconds. What is it?

    hands on

    Not knowing this costs

    Ejecting healthy pods shifts their load onto survivors, which then eject too.

    • Ejection, oscillating on baseEjectionTime
    • Pods never restart and readiness stays green throughout
    • The threshold is below normal application error rates
    shellwatch the counter move
    kubectl exec deploy/checkout -c istio-proxy -n checkout -- \
  pilot-agent request GET stats | grep outlier

# ...outlier_detection.ejections_active: 3        <- benched right now
# ...ejections_enforced_consecutive_5xx: 47       <- and climbing
    The cadence matching baseEjectionTime is the tell. A capacity problem does not recover on a timer. Raise consecutive5xxErrors above what the app hits normally.
  3. 14

    Requests fail under load but the upstream is idle. What is it?

    hands on

    Not knowing this costs

    Scaling the upstream cannot fix requests that are refused before leaving.

    • Flag UO: your own circuit breaker refusing
    • Failure rate scales with concurrency, not upstream latency
    • The pool is sized below real peak
    shellUO is the separator
    kubectl exec deploy/checkout -c istio-proxy -n checkout -- \
  pilot-agent request GET stats | grep overflow

# ...upstream_cx_overflow: 1204          <- connections refused
# ...upstream_rq_pending_overflow: 388   <- queue refused too
    The upstream never saw these requests, which is why its dashboard is clean. Size the pool from observed peak concurrency, then keep a limit, because unbounded pools turn a slow dependency into memory exhaustion.
fig 1Six symptoms, told apart by one detail each
FlagWhat you seeThereforeRung
UHNo subset cluster exists at allRoute names a subset no rule declares12
UHCluster exists, zero endpoints behind itSubset labels match no pod10
UHEndpoint list shrinks and regrows on a timerOutlier detection ejecting healthy pods13
UOoverflow counters climbing, upstream idleConnection pool below real peak concurrency14
200P99 rising, P50 flat, zero errorsPending queue absorbing overload as latency07
200Envoy defaults where your limits should beShort host name, rule bound to nothing11

Three of these show the same flag and the endpoint list separates them. Two show no error at all, which is why the quiet column exists. The rung number is where each one is worked through.

Before you scroll on

0/3

You should now be able to

05

Beyond the basics

15–17

Once it works, the questions become about zones, ownership, and who can even see the rule.

the parts that appear at fleet scale · for whoever owns the platform

  1. 15

    Why did my zone failover drill do nothing?

    hands on

    Not knowing this costs

    The drill passes review and fails the real regional outage.

    • Locality failover only demotes unhealthy endpoints
    • Only outlier detection marks endpoints unhealthy
    • No outlierDetection block means failover never triggers
    manifestthe pair that must travel together
    spec:
  host: catalog.prod.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      localityLbSetting:
        enabled: true
        failover:
          - from: europe-west4
            to: europe-west1
    outlierDetection:            # without this, the block above is inert
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
    The config reads complete without the second block and passes review. A failover that has never actually failed over is a hypothesis, not a control.
  2. 16

    Two teams wrote rules for the same host. Who wins?

    hands on

    Not knowing this costs

    Your settings can stop working because another team applied theirs first.

    • One rule wins outright; policies do not merge
    • Among equals, the oldest by creation time
    • The loser is dropped without any event
    shellfind the collision
    kubectl get destinationrules -A -o custom-columns=\
NS:.metadata.namespace,NAME:.metadata.name,\
HOST:.spec.host,CREATED:.metadata.creationTimestamp

# two rows with one host: the older one is the only one in force
istioctl analyze -A   # reports the conflict outright
    Keep one rule per host with one owner, and express per team needs as subsets inside it. Merge order is not something an on call engineer should reason about.
  3. 17

    Why does the rule work in one namespace and not another?

    hands on

    Not knowing this costs

    A policy that fails for everyone is easy. One that fails per caller wastes a day.

    • exportTo controls which namespaces receive the rule
    • A value of dot means this namespace only
    • The same split comes from a narrow Sidecar resource
    shellthe per caller split, confirmed
    kubectl get destinationrule payment-svc -n payments \
  -o jsonpath='{.spec.exportTo}'
# ["."]        <- visible only inside payments

# same FQDN, two proxies, different answers = this bug
istioctl proxy-config cluster deploy/checkout -n checkout \
  --fqdn payment-svc.payments.svc.cluster.local -o json | grep -c maxConnections
    Verify from a consumer pod in another namespace, never from the rule namespace, because that is the one place this bug is invisible.

Before you scroll on

0/3

You should now be able to

06

Where it ends

18–18

What this file does not decide, which is the part that gets misassigned in reviews.

the limits, where people get caught · for whoever reviews the design

  1. 18

    What does a DestinationRule not decide?

    Not knowing this costs

    Reviews assign these three to this file constantly, and all three are elsewhere.

    • Not where requests go: that is VirtualService
    • Not Kubernetes readiness: ejection is a separate system
    • Not security: anything skipping the proxy ignores it

Before you scroll on

0/3

You should now be able to

Go deeper

5 links, each earning its place.

DestinationRule reference↗

The authoritative field list. Most incidents here trace to a field doing something slightly different from what its name suggests.

Traffic management concepts↗

The division of labour behind rung 3: VirtualService decides where, DestinationRule decides what happens once it gets there.

Circuit breaking task↗

A runnable walkthrough that trips the pool on purpose. Doing it once makes the UO flag from rung 14 instantly recognisable.

Locality failover task↗

The complete working config for rung 15, including the outlier detection block the drills silently die without.

Envoy response flags↗

The definitive table for UH, UO, UF and friends. It turns a generic 503 into a narrow hypothesis in seconds.

Where this leaves you

Rungs 1 to 3 are what most people need. Rungs 9 to 11 are the three commands that belong in your pipeline before any traffic shift.

If you keep one thing: a DestinationRule that binds to nothing looks identical in git to one that works. Only the proxy knows the difference, so ask the proxy.