Stop reading whenever you have enough

Observability, in twelve questions.

Each one is the question the previous answer makes you ask. The ones that matter come with the command, because that is the part you meet at work.

Not technical? The first three questions are written for you, and they are enough to follow any conversation about this.

Rungs
12
Hands on
5
Read
~8 min
01

What it is

01–04

Because every call crosses a proxy, every call gets measured, without anyone writing measurement code.

three minutes, no cluster needed · for anyone

  1. 01

    What do I get without writing any code?

    Not knowing this costs

    Uniform telemetry for every language is the mesh’s quietest superpower.

    • Metrics: every request counted and timed
    • Access logs: every request as a line, if enabled
    • Traces: almost, and the almost is rung 4
  2. 02

    Why can the proxy measure what my app never did?

    Not knowing this costs

    The team that never instrumented anything now has dashboards anyway.

    • Every call enters and leaves through it
    • It sees timing, status and peer identity
    • Measurement becomes infrastructure, not discipline
  3. 04

    Why are traces not free like metrics?

    Not knowing this costs

    This one paragraph is the most misunderstood sentence in Istio.

    • The proxy starts spans, but cannot join them
    • Your code must copy trace headers inward to outward
    • No propagation, no end to end trace, ever

Before you scroll on

0/3

You should now be able to

02

The metrics you get free

03–06

Every request becomes a row with the same labels, for every service, in every language.

the standard series and the labels that matter · for whoever builds the dashboards

  1. 03

    Which two metrics carry everything?

    Not knowing this costs

    Learn two names and every mesh dashboard stops being magic.

    • istio_requests_total: the counter with the labels
    • istio_request_duration_milliseconds: the histogram
    • Rate, errors and duration all come from these
  2. 05

    What are the golden queries?

    hands on

    Not knowing this costs

    Every incident review opens with some dressing of these three.

    • Rate, error ratio, and P99, per service
    • All three from the two metrics, by label
    shellthe dashboard, in three lines
    # traffic
sum(rate(istio_requests_total{destination_service_name="payment-svc"}[5m]))

# error ratio
sum(rate(istio_requests_total{destination_service_name="payment-svc",
  response_code=~"5.."}[5m]))
/ sum(rate(istio_requests_total{destination_service_name="payment-svc"}[5m]))

# p99 latency
histogram_quantile(0.99, sum by (le) (
  rate(istio_request_duration_milliseconds_bucket{
    destination_service_name="payment-svc"}[5m])))
    Prefer reporter="destination" for the truth about what a service experienced, and reporter="source" when you care what its callers saw. They differ exactly when things are interesting.
  3. 06

    Which labels do people ignore until an incident?

    hands on

    Not knowing this costs

    A 503 graph without flags is a mystery. With flags it is a diagnosis.

    • response_flags: why Envoy did what it did
    • connection_security_policy: mTLS or plaintext, per request
    • destination_version: the canary comparison label
    shellthe flag breakdown, live
    sum by (response_flags) (
  rate(istio_requests_total{destination_service_name="payment-svc",
    response_code="503"}[5m]))

# {response_flags="UH"} 2.1   <- cluster exists, no healthy pod in it
# {response_flags="UO"} 0.4   <- circuit breaker: pool too small
# {response_flags="-"}  0.1   <- the app itself said 503
    One query splits a 503 storm into its actual causes. The flags ladder rungs across DestinationRule and Envoy pages all land here.

Before you scroll on

0/3

You should now be able to

03

Proving it works

07–08

The signals are only real once you have pulled them yourself.

commands that answer yes or no · for whoever has to sign it off

  1. 07

    Where do these metrics physically come from?

    hands on

    Not knowing this costs

    Missing dashboards have two causes, and this command tells them apart.

    • Every proxy serves them on port 15090
    • Prometheus scrapes each pod, not a central point
    shellstraight from one proxy
    kubectl exec deploy/payment-svc -c istio-proxy -n payments -- \
  pilot-agent request GET stats/prometheus | grep istio_requests_total | head -3
    When a service is missing from dashboards, this is the first check: metrics exist at the pod and scraping is broken, or they do not and the pod is unmeshed.
  2. 08

    How do I read one access log line properly?

    hands on

    Not knowing this costs

    Logs off by default means your first incident configures logging mid fire.

    • Enable them mesh wide or per workload
    • The default line is positional, so count fields
    shellon, then read
    # meshConfig: accessLogFile: /dev/stdout   (or Telemetry API per namespace)

kubectl logs deploy/payment-svc -c istio-proxy -n payments --tail=1
# [2026-07-26T09:14:02.123Z] "POST /api/charge HTTP/1.1" 503 UO
#   upstream_reset_before_response_started{overflow} - "-" 0 81 2 -
#      ^code ^flag                                          ^dur ^upstream time
    Code is field 5, flags field 6, duration field 12, upstream service time field 13. Those four answer what failed, why, how slow, and whose slowness it was. The caller identity is not in the default format: add DOWNSTREAM_PEER_URI_SAN yourself, or read the principal off the metrics instead.

Before you scroll on

0/3

You should now be able to

04

When it breaks

09–11

The classic failures are silences: missing traces, missing spans, a graph with holes.

the gaps you will actually hit · for whoever gets paged

  1. 09

    Every trace is one span deep. What is it?

    hands on

    Not knowing this costs

    Buying a tracing backend does not buy propagation. Code does.

    • Services are not propagating the trace headers
    • Each hop starts a fresh, orphaned trace
    • The fix is in application code, per service
    shellthe headers your code must forward
    # copy from incoming request to every outgoing call:
#   traceparent            (W3C, the modern one)
#   x-request-id
# legacy b3: x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled

# most frameworks: one OpenTelemetry middleware does it all
    The mesh cannot do this part: only the app knows which outgoing call belongs to which incoming request. This is rung 4, met in production.
  2. 10

    Why does the Kiali graph have missing edges?

    Not knowing this costs

    A missing edge is an unmeshed workload announcing itself politely.

    • An unmeshed pod on one end of the call
    • Or traffic entering outside the mesh path
    • The graph draws proxy reports, nothing else
  3. 11

    What is sampling hiding from me?

    Not knowing this costs

    The trace you need for the weird bug was sampled away by default.

    • Default tracing keeps a small fraction of traces
    • Rare failures fall between the samples
    • Raise sampling temporarily during investigations

Before you scroll on

0/3

You should now be able to

05

Where it ends

12–12

The mesh sees between services. Inside them, it is blind.

the limits, where people get caught · for whoever reviews the design

  1. 12

    What can mesh telemetry never see?

    Not knowing this costs

    The mesh narrows an incident to a service. Your code narrows it further.

    • Inside the process: queries, queues, locks, GC
    • A slow database call looks like a slow service
    • App level telemetry picks up where proxies stop

Before you scroll on

0/3

You should now be able to

Go deeper

4 links, each earning its place.

Istio standard metrics↗

Every metric and label from rungs 3 to 6, defined precisely.

Distributed tracing FAQ↗

The propagation requirement from rungs 4 and 9, in the project’s own words.

Telemetry API↗

Per namespace control of logs, sampling and metrics, the modern way.

Kiali↗

The graph from rung 10 and what its edges and badges actually mean.

Where this leaves you

Rung 5 is the dashboard. Rung 6 is the label that turns a 503 graph into a diagnosis. Rung 9 is the fix for the trace everyone expected to be free.

If you keep one thing: metrics are free because proxies see calls; traces cost because only your code knows which call caused which. Budget accordingly.