MidTroubleshooting

It is Monday morning. Overnight the platform team migrated the container registry. Now about half the pods across three namespaces are stuck in ImagePullBackOff and new deploys will not start. You are on call. What do you do?

What they are really testing: They want to see if you can separate a per pod failure from a fleet wide credential or DNS failure, and whether you can act on a change made by another team without waiting for permission to think.

A real interview question

It is Monday morning. Overnight the platform team migrated the container registry. Now about half the pods across three namespaces are stuck in ImagePullBackOff and new deploys will not start. You are on call. What do you do?

What most people say

drag me

I would tell the platform team their migration broke everything and wait for them to roll it back. It is not my service that is failing.

It reads junior because it treats the incident as someone else's ticket, it produces zero diagnostic information for whoever does fix it, and it leaves the service degraded while the candidate waits for a reply on Slack.

The follow-ups they ask next

  • How would you have caught this before the migration?

    A pre cutover check that pulls every distinct image reference in the cluster from the new registry, plus a canary namespace cut over first. Say you would have asked for that in the migration review.

  • The error says manifest unknown, not unauthorized. Where do you go?

    That is a path or tag problem, not credentials. Check whether the migration copied all tags, whether the repository name changed case, and whether anything still references a mutable tag like latest that was never mirrored.

  • A node reboots mid incident and takes down four healthy pods. What now?

    That is exactly why you froze scaling. Cordon the remaining nodes, and if the pull is still broken, patch the deployment to the old registry reference as a temporary rollback while the real fix lands.

What the interviewer is listening for

  • Realises running pods survive on cached images and protects them by freezing scaling
  • Reads the literal pull error instead of reacting to the state name
  • Tests the image pull by hand from a node to prove the hypothesis
  • Names concrete ECR permissions and imagePullSecret as the likely suspects

What sinks the answer

  • Escalates and waits without gathering a single diagnostic
  • Restarts all deployments, which turns a partial outage into a full one
  • Cannot distinguish unauthorized from manifest unknown
  • Never mentions freezing deploys or autoscaling during the incident

If you genuinely do not know

Say this instead of freezing. Reasoning out loud from what you do know beats silence every single time, and a good interviewer is listening for exactly that.

I have not run a registry migration myself, but here is how I would reason about it. ImagePullBackOff is one of three things, credentials, a wrong path, or networking, so I would first stop anything that could replace a healthy pod, then read the exact pull error and prove which of the three it is.

Keep going with troubleshooting

All 336 cloud engineer questions

Knowing the answer is not the same as recalling it under pressure

Sign in to send the questions you fumble to spaced recall, so they come back right before you would forget them, and learn the concepts behind them with hands-on labs.

Start free