Stop reading whenever you have enough

Multi cluster, in twelve questions.

Each one is the question the previous answer makes you ask. The ones that matter come with the command, because that is the part you meet at work.

Not technical? The first three questions are written for you, and they are enough to follow any conversation about this.

Rungs
12
Hands on
4
Read
~9 min
01

What it is

01–03

Two buildings, one company. This is making services in both behave like one mesh.

three minutes, no cluster needed · for anyone

  1. 01

    What does one mesh across clusters mean?

    Not knowing this costs

    Done right, cluster becomes an implementation detail. Done wrong, twice the outages.

    • A service in cluster A calls one in cluster B
    • Same name, same mTLS, same policies throughout
    • The caller never knows which cluster answered
  2. 02

    What must the clusters share before anything works?

    Not knowing this costs

    Both cables come from ladders you already have: certificates and architecture.

    • One root of trust, or identities cannot verify
    • Discovery: each side must learn the other’s endpoints
    • Everything else is topology detail
  3. 03

    When are two separate meshes the better answer?

    Not knowing this costs

    One mesh is a convenience decision that is also a blast radius decision.

    • Different companies, compliance domains, or blast zones
    • Rare cross traffic through ordinary gateways
    • Federation is a door; one mesh is an open floor

Before you scroll on

0/3

You should now be able to

02

The shapes

04–05

One brain or several, one network or several. Every setup is a cell in that grid.

the two topologies and how to choose · for whoever designs the platform

  1. 04

    What is primary remote versus multi primary?

    Not knowing this costs

    The topology choice is really the control plane outage story.

    • Primary remote: one istiod runs both clusters
    • Multi primary: each cluster runs its own
    • Multi primary survives a whole cluster dying
  2. 05

    What is the east west gateway?

    Not knowing this costs

    On separate networks, every cross cluster byte rides this deployment.

    • A gateway for cluster to cluster traffic
    • Needed when pod networks cannot reach each other
    • Same machinery as ingress, pointed sideways

Before you scroll on

0/3

You should now be able to

03

The wiring

06–07

Three cables join two clusters: shared trust, shared discovery, and a path for traffic.

trust, discovery, and the gateway between · for whoever builds it

  1. 06

    How do two clusters share one root of trust?

    hands on

    Not knowing this costs

    Skipping shared trust at day one is the most expensive multi cluster mistake.

    • Same root signs each cluster’s intermediate
    • The cacerts procedure, run per cluster
    • Different intermediates, one chain, both sides verify
    shellone root, two cacerts
    # per cluster, with per-cluster intermediates from ONE root:
kubectl --context=cluster-a create secret generic cacerts -n istio-system \
  --from-file=ca-cert.pem --from-file=ca-key.pem \
  --from-file=root-cert.pem --from-file=cert-chain.pem
kubectl --context=cluster-b create secret generic cacerts -n istio-system \
  --from-file=ca-cert.pem --from-file=ca-key.pem \
  --from-file=root-cert.pem --from-file=cert-chain.pem
    Two clusters that each generated their own default root can never trust each other retroactively without re rooting one, which is the migration from the certificates ladder, doubled.
  2. 07

    How does cluster A learn cluster B’s endpoints?

    hands on

    Not knowing this costs

    Discovery credentials are the multi cluster secret nobody inventories.

    • A remote secret: read credentials for the other API server
    • istiod watches both, merges endpoint lists
    • Services become one pool across clusters
    shelldiscovery, both directions
    istioctl create-remote-secret --context=cluster-b --name=cluster-b \
  | kubectl --context=cluster-a apply -f -

istioctl create-remote-secret --context=cluster-a --name=cluster-a \
  | kubectl --context=cluster-b apply -f -
    The remote secret is API server read access across clusters: real credentials with a real blast radius, owned and rotated like any other.

Before you scroll on

0/3

You should now be able to

04

Proving it works

08–09

Cross cluster is proven when a call fails over without anyone doing anything.

commands that answer yes or no · for whoever has to sign it off

  1. 08

    Does my proxy actually know the remote endpoints?

    hands on

    Not knowing this costs

    Failover you never saw in an endpoint list is failover you assumed.

    • Read the endpoint list for a service on both sides
    • Remote pods appear behind the east west gateway address
    shellthe merged pool, visible
    istioctl --context=cluster-a proxy-config endpoint deploy/checkout -n checkout \
  --cluster "outbound|8080||payment-svc.payments.svc.cluster.local"

# 10.4.1.9:8080     HEALTHY   <- local pod
# 240.240.0.7:15443 HEALTHY   <- remote, via east west gateway
    No remote entries means discovery or the gateway is broken, in that order. This is the multi cluster version of the endpoint check from the DestinationRule ladder.
  2. 09

    Does failover actually happen?

    hands on

    Not knowing this costs

    Multi cluster resilience that was never drilled is a diagram, not a property.

    • Drain the local side, watch traffic cross
    • Outlier detection required, same as locality failover
    • The drill is the proof, exactly like single cluster
    shellthe drill, cross cluster
    kubectl --context=cluster-a scale deploy/payment-svc -n payments --replicas=0

kubectl --context=cluster-a exec deploy/sleep -n payments -- \
  curl -s payment-svc.payments:8080/version
# v1-cluster-b   <- answered from the other building
    This works only with outlier detection configured: cross cluster failover rides the same health machinery as zone failover, one ladder over.

Before you scroll on

0/3

You should now be able to

05

Where it ends

10–12

One mesh across clusters is one failure domain more often than people admit.

the limits, where people get caught · for whoever reviews the design

  1. 10

    What does latency do to the picture?

    Not knowing this costs

    A call chain that ping pongs between clusters multiplies the distance.

    • Cross cluster hops cost real milliseconds
    • Locality settings keep traffic local until failure
    • Chatty services must not straddle clusters casually
  2. 11

    What still fails together despite two clusters?

    Not knowing this costs

    Multi cluster removes some single points by making others global.

    • Shared root of trust, shared registries, shared config repo
    • A bad mesh wide policy applies everywhere at once
    • Two clusters, one GitOps pipeline: one blast radius
  3. 12

    What do I monitor that single cluster meshes never need?

    Not knowing this costs

    The failure modes you added are the ones the old dashboards cannot see.

    • East west gateway health and throughput
    • Remote secret validity, both directions
    • Cross cluster request share, so drift is visible

Before you scroll on

0/3

You should now be able to

Go deeper

4 links, each earning its place.

Multi cluster installation↗

The topology chooser and the full wiring for each cell of the grid.

Deployment models↗

The primary remote versus multi primary decision from rung 4, formally.

Verify multi cluster traffic↗

The official version of the proofs in rungs 8 and 9.

Locality load balancing↗

The keep it local until failure behaviour rung 10 depends on.

Where this leaves you

Rung 6 is the decision that cannot be retrofitted cheaply. Rungs 8 and 9 are the proofs that the diagram is real.

If you keep one thing: shared trust and shared discovery are the whole game, and both must be decided before the second cluster ships traffic, not after.