Stop reading whenever you have enough

Performance, in twelve questions.

Each one is the question the previous answer makes you ask. The ones that matter come with the command, because that is the part you meet at work.

Not technical? The first three questions are written for you, and they are enough to follow any conversation about this.

Rungs
12
Hands on
5
Read
~8 min
01

What it costs

01–03

Every call pays a small toll at each proxy. Whether that matters depends entirely on your numbers.

the honest bill, in numbers · for anyone defending or attacking the mesh

  1. 01

    What does one hop through a proxy cost?

    Not knowing this costs

    The same toll is invisible to one system and fatal to another.

    • Rough order: a fraction of a millisecond to a few
    • Two proxies per call: yours and theirs
    • Against a 100ms service call, noise. Against 2ms, not
  2. 02

    Where does proxy memory go?

    Not knowing this costs

    A thousand service mesh makes every tiny pod carry a phonebook.

    • Mostly config: every service the proxy knows about
    • Default: every proxy knows the entire mesh
    • Memory grows with mesh size, not pod traffic
  3. 03

    When does the toll actually matter?

    Not knowing this costs

    Most meshes never hit any of the three, and tune anyway.

    • Deep call chains: tolls stack per hop
    • Single digit millisecond latency budgets
    • Very high request rates on small pods

Before you scroll on

0/3

You should now be able to

02

Measuring before touching

04–05

Tuning without a baseline is redecorating in the dark.

the baseline, or nothing else counts · for whoever owns the tuning ticket

  1. 04

    What is the proxy’s share of my latency?

    hands on

    Not knowing this costs

    Without this number, tuning success is unfalsifiable.

    • Compare total duration against upstream service time
    • The difference is the mesh share, per request
    shellthe mesh share, from real traffic
    kubectl logs deploy/checkout -c istio-proxy -n checkout --tail=200 \
  | awk '$12 ~ /^[0-9]+$/ && $13 ~ /^[0-9]+$/ { sum += $12 - $13; n++ }
         END { if (n) print sum/n " ms mesh share avg over " n " requests" }'

# 1.2 ms mesh share avg over 183 requests   <- measured, not estimated
    In the default access log, field 12 is total duration and field 13 is upstream service time, both plain positional numbers. Set a custom accessLogFormat and these column numbers move, which is the usual reason this one-liner prints nothing.
  2. 05

    What are the proxies costing in CPU and memory?

    hands on

    Not knowing this costs

    The mesh bill hides in per pod slices nobody sums.

    • They are containers: kubectl top sees them
    • Sum them: the fleet cost is the real bill
    shellthe fleet bill
    kubectl top pods -A --containers | awk '$3=="istio-proxy" {cpu+=$4; mem+=$5; n++}
  END {print n " proxies, ~" cpu "m CPU, ~" mem "Mi memory total"}'

# 412 proxies, ~8600m CPU, ~19000Mi memory total
    Nineteen gigabytes of phonebooks is the rung 2 problem, fleet wide. It is also the before number for the Sidecar resource fix below.

Before you scroll on

0/3

You should now be able to

03

The two big knobs

06–07

One knob shrinks what every proxy must know. One matches proxy threads to real load. Together they are most of tuning.

the settings that actually move numbers · for whoever operates a large mesh

  1. 06

    How do I shrink what every proxy knows?

    hands on

    Not knowing this costs

    The single biggest memory lever in large meshes, and the most skipped.

    • A Sidecar resource scopes egress per namespace
    • Proxies get config for dependencies, not the world
    • Memory and push load drop together
    manifestthe phonebook, trimmed
    apiVersion: networking.istio.io/v1
kind: Sidecar
metadata:
  name: default
  namespace: payments
spec:
  egress:
    - hosts:
        - "./*"                  # own namespace
        - "checkout/*"           # actual dependencies only
        - "istio-system/*"
    The trap: a dependency missing from this list becomes unreachable, exactly like exportTo hiding. Roll it out namespace by namespace with the egress inventory in hand.
  2. 07

    How do proxy resources get right sized?

    hands on

    Not knowing this costs

    Default proxy sizing is wrong in both directions at once across a fleet.

    • Set requests from measured usage, not templates
    • concurrency matches worker threads to CPU allotment
    • Per pod annotation or mesh wide default
    manifestsizing, per workload where it matters
    # pod template annotations:
  metadata:
    annotations:
      sidecar.istio.io/proxyCPU: "200m"
      sidecar.istio.io/proxyMemory: "192Mi"
      proxy.istio.io/config: |
        concurrency: 2
    concurrency 0 means match the CPU limit, which is right when a limit exists. Two worker threads saturating a 200m request is the common misfit on small pods.

Before you scroll on

0/3

You should now be able to

04

The small knobs

08–09

Telemetry volume and connection reuse: real, smaller, and cheap to try.

the second tier, applied when measured · for whoever is still not done

  1. 08

    What does telemetry volume cost?

    hands on

    Not knowing this costs

    Telemetry is the cost you configured by not configuring it.

    • Every metric label set is series cardinality
    • Every access log line is CPU and storage
    • Trim per namespace with the Telemetry API
    manifestlogs off where they are pure cost
    apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: quiet-hot-path
  namespace: adserving
spec:
  accessLogging:
    - providers:
        - name: envoy
      disabled: true
    A hot path at thousands of requests per second pays real CPU for log lines nobody reads. Keep logs on everywhere humans debug, off where only volume lives.
  2. 09

    Which knobs are usually not worth touching first?

    Not knowing this costs

    Deep tuning before scoping config is polishing the wrong surface.

    • Exotic Envoy filters and hand tuned buffers
    • Protocol tweaks before the two big knobs
    • Anything you cannot tie to the rung 4 number

Before you scroll on

0/3

You should now be able to

05

Where it ends

10–12

At some point the mesh is not your bottleneck, and knowing when is the skill.

the limits, where people get caught · for whoever reviews the effort

  1. 10

    When does proxy tuning stop paying?

    Not knowing this costs

    Past this line, tuning the mesh is avoiding tuning the app.

    • When mesh share is small against app latency
    • When the fleet bill is small against app resources
    • Both are measurements you already have
  2. 11

    What if the toll is genuinely too high?

    Not knowing this costs

    A latency budget the sidecar cannot meet is a design conversation.

    • Ambient mesh moves proxying off the pod
    • Or selectively unmesh the extreme path
    • Both are architecture changes, not knobs
  3. 12

    What do I write down when tuning ends?

    Not knowing this costs

    Undocumented tuning gets re litigated at every capacity review.

    • The baseline, the knobs applied, the deltas
    • The knobs considered and rejected, with numbers
    • The next person inherits conclusions, not folklore

Before you scroll on

0/3

You should now be able to

Go deeper

4 links, each earning its place.

Istio performance and scalability↗

The project’s own numbers behind rung 1, and how they were measured.

Sidecar resource reference↗

The scoping resource from rung 6, every field.

Proxy resource annotations↗

The per pod sizing surface from rung 7, complete.

Ambient mesh↗

The architecture level answer from rung 11, when knobs are not enough.

Where this leaves you

Rung 4 is the number everything else must move. Rung 6 is the biggest lever in a large mesh, and the most commonly skipped.

If you keep one thing: measure the mesh share first. Every tuning decision after that is arithmetic instead of argument.