Your deployment pipeline is green through dev and staging, and fails every time on the production stage. Same commit, same image. How do you debug that?

Question

Accepted Answer

First I stabilise. I freeze the pipeline so we are not stacking half applied deploys, and I confirm production is still healthy on the previous version, so this is a blocked release and not an outage. That changes how fast I have to move. Then I get the real signal. The pipeline UI just shows a red step, so I pull the raw job log and the platform events, for example kubectl describe on the failing pods or the ECS service events, because the pipeline usually just times out waiting for something that never went ready. Then I list the axes on which production differs: the deploy role, the network path to the registry and the API, the secret names, quotas, replica counts and any approval gate. Same commit and same image means the artefact is fine, so the difference has to be environmental, and that narrows it a lot. I test one axis at a time from the production runner. In the last case I hit, staging pulled the image over a public NAT and production had a private endpoint with a registry policy that did not include the new repository, so the pull timed out after 120 seconds and the rollout stalled at 0 of 6 pods ready. I confirmed it in 15 minutes with a single crictl pull from a prod node. The fix was one line of policy. The bigger fix was making staging use the same private endpoint, so the next time it fails in staging on a Tuesday afternoon instead of in production on a Friday night.

Answer

How would you make staging trustworthy without paying for a full production clone?

Talk about parity on the axes that actually break deploys, identity, network path and secrets shape, rather than parity on size and traffic.

Answer

The business wants this shipped tonight. Do you deploy by hand?

Hold the line politely: offer a time box for the real fix, and if you must ship, say you would do it through the pipeline with an override so it stays logged and repeatable.

Your deployment pipeline is green through dev and staging, and fails every time on the production stage. Same commit, same image. How do you debug that?

Your deployment pipeline is green through dev and staging, and fails every time on the production stage. Same commit, same image. How do you debug that?

The follow-ups they ask next

What the interviewer is listening for

What sinks the answer

If you genuinely do not know

Keep going with troubleshooting

Knowing the answer is not the same as recalling it under pressure