You probably know
more than you can
say out loud.

What gets you hired

They are three steps on a ladder. Continuous integration means everyone merges to main often, and every merge triggers an automated build and test run, so integration bugs surface in 10 minutes instead of at the end of a 3 week branch. Continuous delivery adds the guarantee that every passing build is packaged and proven deployable, but a human still decides when to release. Continuous deployment removes that human gate entirely, so a merge that passes goes straight to production. Most teams should stop at continuous delivery until their test coverage, monitoring and rollback are genuinely trustworthy, because the last step converts every test gap into a production incident.

What gets you hired

Because a passing test proves that one artifact works, not that the source works. If I build separately for production, I ship a binary that was never the one I tested, so all 3 earlier stages of testing were about a different thing. That is not theoretical: a transitive dependency resolves to a new patch version, or a base image tagged latest moved overnight, and now the production build differs with an identical commit hash. So the pattern is build once, tag it immutably with the commit sha, and promote that exact artifact through staging to production. Anything environment-specific comes from configuration injected at runtime, not baked in at build time. The speed saving is real but it is the bonus, not the reason.

What gets you hired

I order stages by how fast they give feedback. First lint and unit tests, because they run in under 2 minutes and catch most mistakes without needing infrastructure. Then build the artifact once, tagged with the commit sha, and scan the image for known vulnerabilities right there, so a bad base image never reaches a registry. Then integration tests against real dependencies, which are slower so they should not block the cheap feedback. Then deploy that exact artifact to staging and run smoke tests. Production is a promotion of the same artifact, not a rebuild, gated by either a human approval or an automated canary depending on how risky the service is. I would also make the pipeline fail loudly on flaky tests rather than retrying silently, because a retry that hides flakiness is how teams stop trusting their own suite.

What gets you hired

First I measure per stage, because the bottleneck is rarely where people assume. Typically I find something like 12 minutes of dependency installation that should be cached, a serial chain of stages with no real dependency, and 3 slow integration tests dominating the suite. Then in order: cache dependencies keyed on the lockfile hash, parallelise the independent stages, and shard tests across runners. After that, split the pipeline by feedback need, so a pull request runs lint, unit and build in under 5 minutes and the exhaustive suite runs post-merge. Bigger runners are the last lever, not the first, because they cost money forever and do not fix a serial dependency chain. The target is under 10 minutes for the blocking path, because that is roughly the point where a developer waits rather than switching context and forgetting.

What gets you hired

I would push back on a blanket retry, because a test failing 1 in 5 runs is a defect, and the two candidates are that the test is badly written or that the code has a real race. A retry cannot tell those apart, and if it is the second one the pipeline is now hiding a production bug. It also has a cultural cost: once retries are normal, people stop reading failures at all. What I would do instead is quarantine it immediately, so it runs and reports but does not block, with a named owner and a fix deadline of about 2 weeks, otherwise it is deleted. Quarantine gets the team unblocked today without pretending the problem is solved. Then fix the usual causes: shared state between tests, dependence on wall-clock time instead of a fake clock, and fixed sleeps instead of waiting for a condition. And I would track flake rate per test so it is a visible number, not a vibe.

What gets you hired

I would not port 200 jobs, I would find out how many matter. First discovery: which jobs ran in the last 90 days, who owns them, what they produce. In my experience a set like this is often 40% dead, so that alone removes most of the work, and I would archive rather than delete so nothing is unrecoverable. Then classify the survivors: a large group of near-identical build-and-test jobs that become one template, and a small group of genuinely strange ones, which are usually the risky ones. I build the shared template on 2 services first and get it genuinely good, because every later migration is then configuration rather than engineering. Then migrate incrementally, running both systems in parallel per service and comparing outputs before cutting over, so risk is spread across weeks instead of concentrated in a cutover day. Critically I set a decommission date up front and treat it as real, because otherwise Jenkins runs for another 3 years and we pay for two systems and two mental models.

What gets you hired

A container is a process on the host, isolated by kernel features: namespaces control what it can see, like its own filesystem and network, and cgroups control what it can consume, like CPU and memory. The key difference is that it shares the host kernel, where a virtual machine ships a whole guest OS with its own kernel on top of a hypervisor. That is why a container image is often 50 to 200 MB and starts in under a second, while a VM image is gigabytes and takes a minute or more to boot. The honest trade-off is isolation strength: a shared kernel is a bigger blast radius than a hypervisor boundary, so for untrusted multi-tenant code people run containers inside lightweight VMs to get both.

What gets you hired

Two separate problems, and layer caching fixes both. For build time, the usual cause is copying the whole source before installing dependencies, so every one-line change invalidates the cache and reinstalls everything. Copying just the lock file and manifest first, installing, then copying source typically takes an 8 minute build to under 2 on a warm cache. For size, a multi-stage build: compile in a builder stage with the full toolchain, then copy only the runtime output into a slim final image, so compilers, dev dependencies and build caches never ship. That alone often takes 1.2 GB to around 150 MB. Then a dockerignore so the build context does not include node_modules and git history, and a slimmer base if the runtime allows it. I would measure after each change rather than doing all four blindly.

What gets you hired

It was invented to fix an incentive problem, not a tooling one. Developers were rewarded for shipping change and operations for preventing it, so change queued up into big risky releases handed over a wall. A quarterly release of 300 changes is nearly impossible to debug when it breaks, because any of the 300 could be the cause. DevOps says the people who build a system also run it, which aligns everyone on the same outcome and makes small frequent changes the safe option. The tooling, CI/CD, infrastructure as code, observability, exists to make those small changes cheap and reversible. I would rather join a team that deploys twice a day with modest tools than one with a perfect toolchain and a monthly release train.

What gets you hired

The four DORA metrics, because they balance. Deployment frequency and lead time for change measure speed, change failure rate and time to restore measure stability, and the pairing is what makes them hard to game: you cannot look good by deploying recklessly, because failure rate catches it, and you cannot look good by freezing, because frequency catches that. I measure at team and system level and never per individual, since individual metrics drive people to optimise their own number at the expense of the team outcome. I would add two more: voluntary adoption, because a platform teams are forced onto is not proven, and a periodic developer survey, since a platform can be technically excellent and horrible to use. Then I translate for leadership: lead time is how fast an idea reaches a customer, change failure rate is how often customers feel our mistakes. If lead time drops from 6 days to 2, that is the argument, not a graph of pipeline runs.

What gets you hired

It means the infrastructure is described in version-controlled files that are the source of truth, and a tool reconciles reality to them. Speed is the smallest benefit. The real ones are that a production change becomes a pull request a second person reviews before it happens, that I can stand up an identical staging environment or a second region from the same code in 20 minutes, and that git history answers who changed what and when during an incident. Rollback becomes a revert instead of trying to remember which checkbox someone ticked. The honest cost is that it is slower for a true one-off, and if people still make manual console changes you get drift, where the code and reality quietly disagree until a deploy overwrites something important.

What gets you hired

State is the mapping between the resources in my configuration and the real objects that exist. Terraform needs it to know that this aws_instance block is that specific running instance, so it can compute a diff instead of recreating everything. If 2 people apply at once against the same state, they race: both read the same starting point, both write, and you can end up with a corrupted file or duplicated resources that nothing is now tracking. The fix is a remote backend with locking, so the second apply blocks until the first finishes, plus object versioning so a bad write can be rolled back. The other thing people miss is that state can contain secrets in plain text, database passwords for example, so it must be encrypted at rest with tight access control and must never live in git.

What gets you hired

I do not apply. A plan that destroys a production database means state and reality have diverged, and applying resolves that divergence by deleting data. First I read the plan properly: is it a replace forced by changing an immutable attribute, or a delete because the resource left the config? Then I look at what was changed manually and decide what the intended state actually is. Usually the right move is to update the Terraform configuration to describe the real, intended resource, then run plan repeatedly until it shows no changes, which proves code and reality now agree. If the resource is untracked, terraform import binds it. Then I make the accident hard to repeat: prevent_destroy in the lifecycle block, deletion protection on the database, removing console write access in production, and drift detection running nightly so I hear about this in a report rather than in a scary plan 3 weeks later.

What gets you hired

I think about a state file as three things at once: the unit of locking, the unit of plan time, and the unit of blast radius. That framing answers the question. One state for everything gives 15 minute plans, constant lock contention with 4 teams, and the property that a bad change can propose destroying unrelated production infrastructure, which is the scariest one. The opposite extreme, a state per resource, produces dependency spaghetti where a single logical change needs 12 applies in the right order. So I split by blast radius and change frequency: foundational networking and shared data stores in their own states because they change rarely and failure is catastrophic, then per-service states because those change daily and a mistake should only affect that service. Environments are separate directories with separate backends rather than workspaces, because workspaces share configuration and environment differences end up as conditionals that become unreadable. For cross-state references I prefer data sources on stable identifiers over reading another state directly, since remote state reads couple you to another team internal structure.

What gets you hired

Non-secret config like log level or feature flags is injected from the environment at runtime, so the same artifact runs in all 3 environments. Secrets go in a managed secret store, and the running workload fetches them using its own identity rather than a shared credential. The repository is disqualifying for secrets because git history is permanent: once a key is committed, anyone who ever cloned the repo has it, and deleting the line in a later commit does not remove it, it just hides it from the current view. If it does get committed, the only correct response is to rotate the credential immediately and treat history rewriting as cleanup, not as the fix. Locally I use a gitignored env file, which is fine because it never leaves the machine.

What gets you hired

Merge takes the two branch tips and creates a merge commit, preserving exactly what happened, including that the work was parallel. Rebase replays my commits one at a time on top of the target, creating new commits with new hashes and a straight line of history. The rule I hold to is that rebasing is safe on a branch only I have, and dangerous on a branch anyone else has pulled, because rewriting shared history means their local copy and the remote have diverged and 5 other people now need a recovery procedure. So in practice: rebase my feature branch onto main to stay current and to squash noisy fixup commits before review, then merge that branch into main. That gives a readable history without ever rewriting anything shared.

What gets you hired

For a web app that deploys continuously, trunk-based with short-lived branches. Everyone branches off main, keeps the branch under 2 days, and merges back behind a feature flag if the work is not finished. The reason is integration risk: a branch that lives 3 weeks accumulates conflicts and hides the fact that two people restructured the same module, and you find out at merge time under deadline pressure. Short branches surface that in hours. Feature flags decouple deploying code from releasing behaviour, which is what makes it safe to merge unfinished work. I would pair that with required review, a green pipeline before merge, and protected main. Git Flow with develop and release branches earns its complexity when you ship versioned software and support multiple releases at once, which is not this case, and adopting it here mostly adds ceremony.

What gets you hired

I would start with what a user feels: request rate, error rate, and latency at p95 and p99 rather than the average, because an average of 200ms can hide 5% of users waiting 4 seconds. Then saturation, how close resources are to their limit, since that predicts the next failure. CPU and memory matter for diagnosis but they are bad paging signals, because a service can be at 30% CPU and failing every request. So I alert on user-visible symptoms tied to an objective, for example paging when the error budget for a 99.9% target is burning fast enough to exhaust in hours, and leave resource dashboards for the investigation once someone is already looking. I would also want structured logs and traces so a spike leads to a cause instead of a guess.

What gets you hired

I treat this as a reliability problem, because a team that ignores 40 alerts will ignore the one that matters. First I measure: group by rule and count firings over the last 30 days, plus how many led to any action. Typically 5 rules generate 80% of the volume and almost none of them ever caused someone to do something. Those get deleted, not tuned, because an alert nobody acts on is a metric wearing a costume. Then I convert cause-based alerts into symptom-based ones: instead of paging on each pod restart, page when the user-visible error rate threatens the SLO. Cause metrics stay on dashboards for diagnosis. Then routing by urgency: page for things needing a human within minutes, ticket for things needing action this week, dashboard for everything else. The bar I hold is that every page should be urgent, actionable and user-visible, and if a responder can do nothing at 3am it should not be a page.

What gets you hired

I would define it as the proportion of checkout requests that complete successfully within 2 seconds, measured over a rolling 28 days, because that is what a user actually experiences: a checkout that succeeds in 30 seconds is a failure even though it returned 200. For the target I would resist reaching for nines and work from consequence: what does an hour of failed checkout cost in revenue and trust, and what would each additional nine cost in architecture and on-call. For most businesses that lands around 99.9%, which is roughly 43 minutes a month of error budget. Framing it as a budget is the useful part, because it converts reliability from an argument into an allowance: while we are inside budget the team ships freely, and when we exhaust it, work shifts to reliability by prior agreement. I would measure at the edge or client side rather than only server side, since server metrics cannot see requests that never arrived, which are exactly the ones users complain about. And it needs an agreed consequence on breach, or it is decoration.

What gets you hired

The pod is the unit that actually runs, one or more containers sharing a network namespace and storage. The important property is that it is disposable: it dies on a node drain, a crash, or every deploy, and comes back with a different IP. Everything else exists because of that. A Deployment declares the desired state, say 3 replicas of this image, and through a ReplicaSet it keeps that true, replacing dead pods and doing rolling updates so old and new versions overlap during a release. A Service solves addressing: since pod IPs churn, it provides one stable DNS name and virtual IP and load balances across the pods matching its label selector. Labels are the connective tissue, and a selector that does not match the pod labels is the single most common reason a service returns nothing.

What gets you hired

Liveness asks whether this container should be restarted. If it fails past the threshold, the kubelet kills and restarts it, so it should only check whether the process itself is wedged. Readiness asks whether this pod should get traffic. If it fails, the pod is pulled out of the Service endpoints but keeps running, so it is right for warming up or shedding load temporarily. The classic disaster is checking a downstream dependency in liveness: the database has a 30 second blip, all 10 pods fail liveness simultaneously, and the entire deployment enters a restart loop, turning a brief dependency hiccup into a full outage that outlives it. That check belongs in readiness, where pods stop taking traffic and then recover on their own. For slow-starting apps I use a startup probe so liveness only applies after boot.

What gets you hired

Requests decide scheduling, limits decide enforcement, and the two resources behave differently when exceeded. Memory is incompressible: cross the limit and the container is killed with exit 137. CPU is compressible: cross the limit and you are throttled, which looks like unexplained latency rather than a crash. So I set the memory request from observed steady state with maybe 20% headroom and set the memory limit close to it, since memory that runs away should be killed rather than take down a node. For CPU I set a request from real usage but I am cautious about tight CPU limits, because throttling produces p99 latency spikes with no error in the logs, which is genuinely hard to diagnose. I get the numbers from actual load data over a week, not from a guess, and revisit after traffic changes. Wrong in one direction wastes money, wrong in the other causes 3am pages.

What gets you hired

Namespace per team is the starting point, with RBAC scoped so a team administers its own namespace and nothing else. Then the parts people forget. Resource quotas and limit ranges, otherwise one team deploying a runaway job consumes the cluster and the other 3 teams have an incident they did not cause. Network policies with a default deny, because Kubernetes networking is flat by default and any pod can reach any other pod across namespaces, which surprises people. Pod security standards so nobody runs privileged containers. Then the honest part: namespaces are an organisational boundary, not a hard security boundary. Tenants share a kernel, share nodes, and cluster-scoped resources like CRDs and webhooks are global, so a container escape or a greedy webhook crosses namespaces. So I would split into separate clusters when tenants are mutually untrusted, when compliance requires demonstrable isolation, when teams need different upgrade cadences, or when the blast radius of one cluster failing is unacceptable. Otherwise multiple clusters mostly multiply operational cost.

What gets you hired

First I read the actual error rather than assuming it is environmental, because a decent share of these turn out to be real bugs the local run never hit, like a test that depends on file ordering. If it is genuinely environmental, I enumerate the differences systematically: dependency versions, since a developer with a warm node_modules can be 3 patch versions behind the lockfile, then environment variables, case sensitivity because macOS is case-insensitive and the CI Linux image is not, timezone, memory limits, and whether the build has network access. The fastest route is usually to run the CI container image locally with the same command, so I am debugging the same environment rather than guessing. Once I find it, I make it impossible to recur: pin the version, containerise the build so local and CI are identical, or move the missing variable into config as code.

What gets you hired

First find where it went rather than deleting blind: du down the tree to find the biggest directories, which usually lands on logs, an unrotated file, a core dump, or a build cache. I also check inodes, because a filesystem can show free space and still fail writes if millions of small files exhausted the inode table, and that presents identically to a full disk. To reclaim, I rotate and compress logs rather than deleting them, since the log that filled the disk is often the evidence for why. The trap to know is that deleting a file a process still holds open frees nothing: the space returns only when the process closes it, so you delete a 40 GB log, df still says 100%, and you go looking for a second problem that does not exist. Check with lsof for deleted-but-open files and restart or signal the holder. Then the actual fix: log rotation configured properly, a retention policy, and an alert at 80% so next time it is a ticket at 2pm rather than a page at 2am.

What gets you hired

Everything failing simultaneously at midnight is a certificate expiry until proven otherwise, because that is one of very few faults that hits a hard time boundary across an entire fleet. I confirm in one command by inspecting the served certificate and reading the notAfter date, which takes 15 seconds and removes all speculation. Then restore: renew and deploy, or if renewal is slow, fail over to a path with a valid certificate. Something people miss is checking every layer, since a system can have an edge certificate, internal service-to-service mTLS, client certificates and pinned chains, and renewing the edge one while an internal one is also expiring means a second outage an hour later. Then prevention, which is the real answer, because this is 100% preventable: automated renewal with something like cert-manager or ACME, monitoring that alerts 30 days before expiry on every certificate including internal ones, and an inventory so no certificate is unowned. If a human remembering is your renewal process, this will recur.

What gets you hired

CrashLoopBackOff means it already restarted several times and Kubernetes is now backing off, so restarting again tells me nothing. I start with logs for the previous instance, since the current container is usually dead: kubectl logs pod --previous. If the app logged a stack trace, I am done in 30 seconds. If logs are empty, the container probably never started, so kubectl describe pod for the events, which surfaces ImagePullBackOff, a missing secret or configmap, or an OOM kill. The exit code narrows it further: 137 is a kill signal, almost always the memory limit, while 1 is the app exiting on its own. After a deploy, the top causes are a missing environment variable, a secret that does not exist in that namespace, and a memory limit that is fine in staging but not under production load. Meanwhile I roll back to the last good revision so users are not waiting on my investigation.

What gets you hired

A 502 means something proxied to my service and did not get a usable response, so I work the path from the outside in. Running is not ready, so first I check whether pods are passing readiness, because a pod can run happily while failing readiness and therefore never join the Service. Then I check the endpoints object directly: if it is empty, that is the answer, and the cause is either a label selector that does not match or every pod failing readiness. If endpoints look fine, I check ports, since a targetPort of 8080 against an app listening on 3000 gives exactly this, as does an app bound to 127.0.0.1 rather than 0.0.0.0, which works when you exec in and fails from outside. Then I port-forward straight to one pod and curl it: if that works, the problem is routing, if it does not, the problem is the app. That splits the search space in half in about 20 seconds.

What gets you hired

NotReady means the kubelet stopped reporting to the API server, and importantly that does not tell me whether the node is dead or just unreachable. Pods hang in Terminating because graceful deletion waits for the kubelet to confirm the container stopped, and an unreachable kubelet never will. My first priority is capacity: after the eviction timeout, usually about 5 minutes, the controller marks pods for eviction and they should reschedule, so I check that replacements are actually landing and that the cluster has room. Then I diagnose the node: disk pressure, memory exhaustion, a crashed kubelet, or a network partition to the control plane are the usual four. If it is genuinely gone, I drain and remove it and let autoscaling replace it. I am careful with force delete, because it removes the pod object without confirming the container stopped, and for a StatefulSet that risks two instances writing to the same volume at once.

What gets you hired

My first move is not a terminal, it is 2 questions: what is the scope, and what changed. Scope means is it every endpoint or one, every region or one, all users or a subset, because "checkout fails for 5% of users" and "nothing responds anywhere" are completely different investigations. Then recency: deploys in the last 4 hours, config or flag changes, infrastructure changes, and certificate expiry, since a very high share of outages are something a human did recently. In parallel I glance at the dependency dashboard and the cloud provider status page so I am not debugging someone else outage. I post in the channel immediately, even just "site down, checking scope and recent deploys", because that prevents 3 other people starting the same work. The moment I have a plausible candidate, usually a recent deploy, I mitigate rather than confirm: rolling back a suspect deploy costs 2 minutes, and being right about the cause before acting can cost 40.

What gets you hired

IP working and name failing isolates it to DNS, which is genuinely useful because it removes routing, firewalls at the IP level and the remote service from suspicion. The fact that it is one namespace is the second clue, because cluster DNS is shared, so a broken resolver would break everything. That points hard at a network policy: a default-deny egress policy in that namespace that allows the API port but forgets UDP port 53 to the DNS service gives exactly this symptom, and it is probably the most common cause I have seen. Otherwise I would resolve from inside an affected pod rather than from my machine, check the pod resolv.conf for search domains and the ndots setting, since a high ndots value means a name gets tried against 4 or 5 search suffixes before the real one and can time out, and verify the DNS pods are healthy and not rate limited. Hardcoding the IP is not a fix, it is a scheduled outage for whenever the provider rotates it.

What gets you hired

First I ask what the failure actually is, because the error is data: connection refused means something answered and said no, so the path works and the target is not listening. A timeout means packets are disappearing, which points at policy or routing. A DNS error means the name never resolved. A TLS error means the connection worked and the handshake failed. Those are four different investigations, so I would not proceed until I know which. Then I test from inside pod A rather than my laptop, resolving the name first and then connecting directly to the pod IP, which cleanly splits DNS from connectivity in about 20 seconds. If the IP works, it is DNS or the service object. If neither works, I check whether B is actually serving: endpoints populated, readiness passing, and the process bound to 0.0.0.0 rather than 127.0.0.1, since binding to localhost works when you exec in and fails from anywhere else. Then network policy on both sides, remembering that egress from A and ingress to B are separate rules.

What gets you hired

Normal resource usage with bad latency means the process is waiting rather than working, so I stop looking at consumption metrics and start looking at queues and blocking. The most common cause by far is a saturated connection or thread pool: a pool of 10 database connections with 50 concurrent requests means 40 requests wait, while CPU shows 20% because nothing is computing. Next, dependency latency, since time spent waiting on a database or third party never appears in host metrics at all, and this is exactly what distributed tracing is for. Then contention: database lock waits or an application-level mutex serialise work no matter how much capacity exists. Then the sneaky ones: CPU throttling against a cgroup quota, which averages out to low utilisation while causing latency spikes within each period, and network or IOPS limits on the instance type. I would get a trace of a slow request first, because it usually points straight at whichever of these it is instead of me checking all five.

What gets you hired

The important thing first: this is Kubernetes working correctly. It will not remove old pods until new ones are ready, so users are still being served and I have time to think rather than panic. Then I look at where the new pod stopped, and there are 3 distinct places. Pending means it never scheduled, and describe will say why: insufficient CPU or memory across nodes, a node selector or affinity nothing satisfies, a taint without a matching toleration, or a volume that cannot attach because it is still bound elsewhere. If it scheduled but is not running, the usual causes are an image that cannot be pulled, often a typo or a missing registry credential, or a missing configmap or secret in that namespace. If it is running but never ready, that is a readiness probe failing, which is either a genuinely broken new version or a probe with too short a timeout for a slower startup. Given old pods are serving fine, I would roll back if the cause is not obvious within about 5 minutes, and investigate without the pressure.

What gets you hired

No deploy and no traffic change points at accumulating state or at something outside my service. First I look at the shape: a step change at a specific minute suggests an event, like a managed database failing over or a config flip, while a gradual ramp over hours suggests something filling up. I also check whether only p99 moved or the whole distribution, since p99 alone often means a subset of requests hitting a bad path, for example one shard or one cache miss pattern. Then tracing to localise: compare a trace from before and one now, and usually 90% of the added time sits in a single span. The classic causes with flat traffic are a table crossing a size where a query stops using an index, a cache reaching capacity so hit rate falls, a queue slowly backing up, or a disk filling and slowing writes. I would also ask whether a dependency deployed, because my no-deploy is not their no-deploy. Scaling up is a last resort, since if the bottleneck is a shared lock, more instances make it worse.

What gets you hired

The whole game is finding what is different about the failing 2%, so I slice by dimension until the failures concentrate. Instance first, because 2% is suspiciously close to 1 bad pod in a fleet of 50, and if all failures come from one instance I have my answer in 30 seconds. If not, then region, service version during a partial rollout, endpoint, customer or tenant, and payload characteristics. Data-dependent failures are common and invisible in aggregate: one customer with 10000 items in a list, or a record with a null nobody expected. Resource exhaustion also presents this way, since a connection pool at its limit fails a small share while most requests get a connection, and that will grow. Then I get one complete trace and one full stack trace of an actual failure, because a single concrete example beats hours of dashboards. I would not accept 2% as within budget: partial failures are usually the early phase of total ones, and 2% of a million requests is 20000 broken experiences.

What gets you hired

A 6 hour cycle with a clean restart is a leak until proven otherwise, but I would rule out the boring explanations first. Is the memory limit simply too low for current traffic, and is the runtime aware of its cgroup limit, because a JVM or Node process that sizes its heap from host memory rather than the container limit will happily grow past it and get killed. Then I look at the growth curve: a steady climb tracking request count means something is retained per request, while growth that continues when traffic drops to nothing points at a background job or an unbounded cache. The decisive evidence is heap profiles captured at 1 hour and 5 hours and diffed, which shows what class of object is accumulating rather than leaving me theorising. The usual culprits are an unbounded cache with no eviction, event listeners never removed, or connections not returned to a pool. I would set a scheduled restart as a stopgap so nobody is paged, but I would say plainly that it is a stopgap with an owner and a date, not a fix.

What gets you hired

I would not scale first, because 3 of the 4 common causes get worse when you do. First: is the arrival rate up or is per-message processing time up, since those are different problems and the graphs distinguish them immediately. Second: are consumers actually healthy, because a crash-looping consumer produces exactly the same queue-depth curve as a slow one, and scaling replicas of a crashing consumer accomplishes nothing. Third: is there a poison message being redelivered forever, which both blocks progress and consumes capacity, and the fix is a dead letter queue with a redelivery limit rather than more workers. Fourth: what are consumers blocked on, because if each one is waiting on a database at its connection limit, doubling consumers doubles contention and total throughput can actually fall. Only once I know the bottleneck is genuinely consumer capacity do I scale. If the backlog is already 6 hours deep and growing, I would also discuss shedding or prioritising, because processing stale messages in strict order can be worse than dropping some.

What gets you hired

I would treat the divergence as the finding, not just the bug. The axes are usually data, concurrency, configuration, scale, and mocked integrations. Data is the most common: staging often has 10000 rows where production has 40 million, so a query that uses an index in one does a table scan in the other, and edge-case records that accumulated over 5 years simply do not exist in staging. Concurrency is next, because tests run sequentially and never produce the race that appears when 200 requests hit the same row. Then configuration, where a flag or a limit differs, and mocked third parties that always return quickly and never fail. To reproduce meaningfully I want production-like conditions: a sanitised production data sample, load at real concurrency, or shadow traffic mirrored to a staging deploy. Then the real fix is closing whichever axis caused it: seed staging to a realistic scale, or add a load test at production concurrency to the pipeline. Otherwise I have fixed one bug and kept the false confidence.

What gets you hired

The absence of a log is the first clue, not a reason to close the ticket. It usually means one of three things: an exception is being caught and swallowed somewhere, the failure is client-side and never reached my server, or the code path has no instrumentation at all. All three are findings. So I start by getting specifics from the user, especially an exact timestamp and their account, because that converts an unsearchable problem into a lookup: I find their actual requests in that 2 minute window and read the full trace, including successful ones, since a 200 response with wrong content is invisible to error monitoring. Then I ask what is different about them: feature flag state, data shape, permissions, client version, region, network quality. If I still cannot see it, I add targeted instrumentation, logging on that specific path and client-side error reporting, and wait for the next occurrence, which usually comes within days. I would tell the user honestly that I cannot see it yet and am instrumenting, because silence makes people stop reporting bugs and that is a much worse outcome.

What gets you hired

Three occurrences after 2 postmortems is a process failure, and I would say that plainly rather than starting a third investigation. So I read the previous 2 first and check the most likely explanation: were the action items actually completed. In my experience this is the answer most of the time, because actions get agreed in the emotional aftermath, no owner or date is attached, and 2 weeks later everyone is on roadmap work. If they were completed, then the second question is whether they were the right kind of action. Items like "be more careful with migrations" or "document the process" feel like prevention and change no system behaviour, so the incident recurs with a different person. Real actions change what is possible: a guardrail, an automated check, a removed permission. Third possibility is that these 3 incidents share a deeper architectural cause each postmortem treated locally. The process fix is that action items get an owner and a date and are tracked like any other work, and a repeat incident automatically escalates to a wider review rather than another copy of the same document.

What gets you hired

The mechanism is resource exhaustion propagating backwards. My service calls a dependency that slows from 50ms to 8 seconds. Each in-flight request now holds a thread and a connection for 8 seconds instead of 50ms, so at any real request rate my pool is exhausted within seconds, and now my service cannot serve any request, including the 90% that never needed that dependency. Then my callers do the same thing, and it walks up the graph. Counter-intuitively a dependency being completely down is safer than being slow, because a refused connection frees the resource instantly. The controls, in order of value: aggressive timeouts on every network call, because unbounded waiting is what enables the whole cascade, then bulkheads so each dependency gets its own bounded pool and cannot starve the others, then circuit breakers to stop sending traffic to something already failing, then graceful degradation so the 90% of the response that does not need it still works. Retries need care: naive retries amplify load on the failing thing, so they need exponential backoff, jitter, and a budget capping retries as a fraction of traffic.

What gets you hired

Rolling replaces instances in batches, so it needs no extra capacity but runs both versions at once and rolls back slowly. Blue-green keeps two full environments and flips traffic in one move, which makes rollback a 30 second switch, but you pay for double capacity and it does not protect you from a bad release, it just makes undoing it fast. Canary sends a small slice, say 5%, to the new version and widens only if error rate and latency hold, so it limits blast radius and actually detects problems rather than just reversing them. I choose on detection speed and blast radius: a high-traffic user-facing service justifies canary because 5% gives a usable signal in minutes. A low-traffic internal tool does not, since that slice would take hours to say anything, so rolling with a fast rollback is more honest. Both versions overlapping also means the database schema has to be compatible with both, whichever I pick.

What gets you hired

Roll back first, diagnose second. The deploy is the obvious suspect and reverting is the fastest path to a known-good state, so if it is behind a feature flag I turn the flag off, which takes seconds, otherwise I redeploy the previous artifact. I say what I am doing in the incident channel as I do it, so nobody else makes a conflicting change. Then I confirm with the error rate actually returning to baseline rather than assuming, because if errors persist after rollback my assumption was wrong and it may be a dependency, not my change. Before cleaning anything up I capture logs, traces and the failing image tag, since that evidence disappears. Only then do I investigate properly, and the output is a blameless write-up plus one concrete change, usually a test or an alert, that would have caught it before 4pm.

What gets you hired

The best answer is to not store a credential at all. Modern CI supports identity federation, so the pipeline presents a signed token proving which repository and branch it is, and the cloud exchanges it for credentials that expire in about 15 minutes. Nothing long-lived exists to leak. If that is not available, then a dedicated deploy identity scoped to exactly what deployment needs, never a personal token, and bound to protected branches and a protected environment so a pull request from a fork cannot reach production credentials. Then the hygiene: masked in logs, never passed as a build argument since those persist in image history, and rotated automatically. I would also want an audit trail of which run used which credential, because during an incident the first question is what had access and when.

What gets you hired

I triage rather than treating 200 as one number. First severity crossed with reachability: a critical in a library my code actually calls on a network path is urgent, a high in a package that ships in the base image but is never loaded is not. Then I look for the cheap win, which is usually the base image, since a base that is 8 months old typically accounts for most of the count and rebuilding on a current one can remove 150 of them in a single change. For what remains I check whether a fix exists at all, because some have no patch and need a compensating control or an accepted risk with an expiry date. Then the policy: the pipeline fails on new critical or high findings so the number cannot grow, while existing ones get owners and deadlines. Blocking everything on day one guarantees someone adds a bypass flag, and then the gate is decorative.

What gets you hired

The mistake is treating it as a swap. Changing the password in place invalidates every live credential at once, so all 12 services fail until they restart, which is a self-inflicted outage. Instead I make old and new valid simultaneously: either the database supports 2 active passwords, or more portably I create a second user with identical grants. Then I roll services onto the new credential one at a time, or in small batches, watching connection errors after each, so if something breaks I have changed one service and can revert it. Once all 12 are migrated I verify with connection or audit logs that the old credential has had zero use for a day, because there is always a forgotten cron job or a scaled-to-zero service that nobody remembers. Then revoke it. Then automate the cycle so this runs on a schedule, because the reason credentials sit unrotated for 3 years is that rotation is a scary manual event.

What gets you hired

The reason it is admin is that narrowing by guesswork means breaking deploys until you find the missing permission, and nobody has the appetite for that twice. So I would derive it from evidence instead. Cloud providers record which API calls an identity actually made, so I take 90 days of that data and generate a policy from real usage rather than from imagination. That gets me most of the way in an afternoon rather than weeks. Then I narrow in stages: apply the derived policy in a development account first, run the full deploy path, and watch for denials in the logs. I would also split the role by job, because the plan step only needs read access and only apply needs write, and giving pull request builds a read-only role removes a large amount of risk immediately. Crucially I set up alerting on access denied events for that role, so instead of fearing a missing permission I find out in seconds with the exact call named. That turns it into a feedback loop rather than a gamble, which is what makes it finishable.

What gets you hired

I would separate the control intent from the ritual usually attached to it. The requirement is normally separation of duties plus an audit trail, and a reviewed pull request satisfies both: a second engineer approved the change, and git records who, what and when permanently, which is stronger evidence than a meeting minute. So the design is protected branches with required review, a pipeline that emits immutable evidence for each deploy, commit sha, approver, tests passed, scan results, artifact digest, and automated policy gates that enforce the actual rules, such as no deploy without a passing scan. That converts compliance from an attestation into an enforced property, which auditors generally prefer once you show them. The critical move is bringing the auditor in early and agreeing what evidence looks like, rather than building it and arguing afterwards. A change advisory board would take us from 10 deploys a day to 1 a week, and a release carrying 50 batched changes is far harder to debug than 1, so the literal reading actively harms the outcome the control exists to protect.

What gets you hired

The reframe is that the pipeline is production: it has deploy credentials and runs third-party code, so it deserves the same controls. Concretely, in order of value. Pin everything by digest rather than tag, both base images and dependencies, because a mutable tag means the thing you reviewed and the thing you built are not necessarily the same. Treat CI actions as code: pin them to a commit sha rather than a version tag, and allowlist which third-party actions can run at all, since an action is arbitrary code holding your credentials. Sign artifacts at build and verify signatures at admission, so an image that did not come from my pipeline cannot run in the cluster even if someone pushes it to the registry. Generate an SBOM per build and store it, because when the next widely-used library has a critical CVE, the difference between answering in 10 minutes and 3 days is whether you already know what you shipped. And remove long-lived credentials in favour of short-lived federated ones, so a compromised run cannot be replayed later.

What gets you hired

During normal running it spreads requests across healthy targets, but during a deploy the interesting parts are health checks and draining. A new instance only joins the pool once it passes its health check, which is why the check must reflect genuine readiness: if it returns 200 the moment the process starts but the app needs 15 more seconds to warm connections, the balancer sends real traffic into errors. When an old instance is removed it should go into draining, so no new requests arrive but in-flight ones get up to say 30 seconds to complete, instead of being cut mid-response. Most deploy-time request drops trace back to one of those two, either a health check that lies about readiness or a shutdown that does not drain. The app also has to handle the termination signal by refusing new work while finishing current work.

What gets you hired

The browser checks its own cache, then the OS, then asks a resolver, which walks from root to TLD to the authoritative server unless something is cached, returning an IP. Then a TCP handshake to that address, 3 packets, followed by TLS: client hello, certificate presented, chain validated against trust stores, key exchange, and ALPN negotiating HTTP/2. That address is usually a CDN or edge proxy, which may serve from cache and never reach me at all. Otherwise it forwards to a load balancer, which picks a backend that is passing health checks. In a cluster that lands on an ingress controller, which matches host and path rules to a service, and the service resolves to a ready pod through its endpoints, which is where readiness probes matter. Packets reach the container through the CNI, the app processes the request, probably calling a database and a cache, and the response returns back along the same chain. Roughly 8 layers, and the useful thing about knowing them is that debugging becomes bisection: identify which layer the failure sits in rather than guessing.

What gets you hired

Refusing connections with healthy resources means a countable limit, not a capacity one. First file descriptors, since every socket is one and the default soft limit is frequently 1024, which a busy service passes easily, and the symptom is "too many open files" in logs. Then the listen backlog: if the app accepts connections slower than they arrive, the accept queue fills and the kernel starts refusing, which looks like the service being down while the process is fine. Then ephemeral ports on whichever side initiates: a service making thousands of short-lived outbound connections exhausts the ~28000 port range, especially with sockets sitting in TIME_WAIT for 60 seconds. Then application-level pool maximums, which are configured numbers people forget exist. Then the infrastructure ones that are invisible from inside the box: conntrack table limits on stateful firewalls, and NAT gateway or load balancer connection ceilings, both of which drop silently. I would check these in that order because it runs cheapest to hardest, and the first two cover most cases.

What gets you hired

You cannot rename atomically, because during rollout both versions of the code are live and a rename breaks one of them. So I use expand and contract over several releases. Release 1 adds the new column, nullable, and deploys code that writes to both columns but still reads the old one, which is safe for both old and new pods. Then I backfill in batches, maybe 5000 rows at a time with a pause, so I do not lock a big table or flood replication. Release 2 switches reads to the new column while still writing both, so rollback is still trivial. Release 3, once I am confident, stops writing the old column. Release 4, after a safe interval, drops it. It is 4 deploys instead of 1, which feels slow, but every step is independently reversible and no step requires downtime or a window.

What gets you hired

The arithmetic is the answer: 100 pods with a pool of 20 each is 2000 connections, against a database typically configured for a few hundred. Connections are not free, since each consumes memory and in Postgres a backend process, so raising max_connections to 2000 trades refused connections for memory exhaustion, which is a worse failure. The right fix is a connection pooler in front of the database, in transaction pooling mode, which multiplexes thousands of client connections onto perhaps 50 real ones. That works because pods hold connections idle most of the time, so the real concurrency is far lower than the connection count. Alongside that I would right-size the per-pod pool, since 20 is usually inherited from a default and a pod serving modest concurrency may need 5. The deeper question is whether scaling to 100 pods was right at all: if the bottleneck was already the database, adding pods increases contention and total throughput can fall, so the honest answer might be fewer pods plus query optimisation rather than more connections.

What gets you hired

First I would ask what staleness is actually acceptable, because that is a product decision and it determines everything. A product catalogue tolerating 5 minutes and an account balance tolerating zero need different designs. Then the strategy: TTL alone is simple but guarantees staleness for the window, while explicit invalidation on write is fresher but fails whenever a write path forgets, which is the usual root cause of exactly this bug. So I would go looking for write paths that bypass the cache-aware code: batch jobs, admin tools, database migrations, and another service writing the same table are the classic four. My default is short TTL as a safety net plus explicit invalidation for freshness, so a missed invalidation is bounded rather than permanent. Then the failure modes caching introduces: stampede, where a popular key expires and 500 requests hit the database at once, fixed with jittered TTLs and single-flight so only one caller recomputes. And cold start after a deploy, where an empty cache means full database load at the worst moment.

What gets you hired

GitOps is continuous reconciliation, not just storing YAML in git. An agent inside the cluster watches a repository and constantly makes actual state match declared state. The difference from a pipeline running kubectl apply is what happens after: push applies once and stops, so if someone edits a deployment by hand at 2am, that change survives indefinitely and nothing notices. A reconciling agent detects the divergence within about a minute and puts it back. Three concrete wins follow. Drift self-heals. CI never needs cluster credentials, because the pipeline only writes to a repository and the agent pulls, which removes production credentials from a system that runs arbitrary code. And the repository becomes a truthful log of what is deployed, so "what is running in production" is a git query rather than an investigation. The costs are real: another controller to run, and a slightly less direct feedback loop for the developer waiting on a deploy.

What gets you hired

I would find the specific line item rather than asking people to be careful. First a month-over-month diff grouped by service, which usually localises 40% to one or two services immediately, then drill to individual resources. The key question is rate versus usage: a bigger instance type is a rate change, 3 times the volume is usage. Common causes with flat traffic are an orphaned resource nobody deleted, unattached storage volumes left behind by a test, a change in data-transfer pattern such as traffic newly crossing availability zones or leaving through a NAT gateway instead of a VPC endpoint, and log or metric volume exploding after someone raised a log level to debug and never lowered it. Then I correlate the daily cost curve with the deploy and infrastructure change history, because the day the curve bent usually matches a change. Afterwards: enforced tagging so spend maps to an owner, anomaly-based budget alerts rather than monthly surprises, and a scheduled sweep for orphaned resources.

What gets you hired

I ran a database migration during business hours that I believed was non-blocking. It took a lock on a large table and the main API started timing out within about 90 seconds. I killed the migration immediately, which released the lock and recovered service in roughly 4 minutes, and I posted in the incident channel as I did it so nobody else started changing things. Total impact was about 6 minutes of elevated errors for roughly a third of requests. The root cause was that I tested on a staging table with 10000 rows and production had 40 million, so the lock behaviour was completely different. In the review I owned it directly, and the fixes were systemic rather than personal: migrations now run through a tool that refuses long-running locks, staging seeds are sized within an order of magnitude of production, and schema changes go through a review checklist. I would rather tell you about this one than a smaller one, because the guardrails came from it.

What gets you hired

A developer wanted to push a hotfix straight to production skipping the staging deploy, because a customer was blocked and staging took 20 minutes. My concern was that the change touched auth and an untested auth change is a much bigger blast radius than the original bug. I started by acknowledging the real cost, since 20 minutes with a customer waiting is genuine. Then I made it concrete rather than procedural: I asked what happens if this is wrong, and the answer was every user is locked out rather than one customer being blocked. That reframed it for both of us. We agreed on a middle path: deploy to staging but only run the auth smoke tests instead of the full suite, which took 4 minutes, and shipped behind a flag so we could disable it instantly. It went out in about 10 minutes total. Afterwards I took the 20 minute staging deploy as my problem to fix, because the process created the pressure to skip it.

What gets you hired

I was paged for high latency on a service I had never touched, at 3am, 2 weeks into the rota. I spent about 40 minutes convinced it was the database because that was the pattern I knew, and I was wrong. The actual cause was a downstream service returning slowly and my service having no timeout, so threads piled up. What went badly was that I anchored on my first theory and did not escalate, partly because I did not want to wake someone. When I finally called the service owner at around 4am, he identified it in 5 minutes because he recognised the shape immediately. What I learned was about myself: I treated escalation as an admission of failure, when the actual cost was 40 extra minutes of user impact. Now I set myself a rule, if I have no confirmed hypothesis after 20 minutes on an unfamiliar service, I escalate. We also added a runbook for that service and a timeout, which was the real defect.

What gets you hired

I spun up a GPU cluster for a load test, and my cleanup script only removed the instances, not the attached storage or the load balancer. That cost about 2800 dollars over 5 weeks before anyone noticed, and what noticed was the monthly bill review, not any system I had built. I owned it directly in the team channel rather than waiting to be asked, because the awkward part is the delay, not the mistake. Root cause was two things: my teardown did not match my setup, and there was no cost anomaly alerting at all, so a 2800 dollar drift was invisible for over a month. The fixes were both systemic. I moved the test environment to Terraform so teardown is a destroy of the whole thing rather than a script I wrote by hand, and I added cost anomaly alerting on daily spend, which caught an unrelated 400 dollar mistake by someone else 6 weeks later. That second catch is the part I am actually pleased about.

What gets you hired

I inherited a billing job with no documentation and an author who had left. My first move was not to read the code but to make it observable, because the immediate risk was it failing silently. I added logging around each phase and an alert on completion time, which told me within a week that it normally ran 40 minutes and occasionally 3 hours, which nobody knew. Then I learned from history rather than documentation: git log showed which parts changed most and therefore where the pain was, and the incident channel search showed 4 past failures with the same signature. I wrote down everything I learned as I went, which became the runbook nobody had. Then small reversible changes: I added a dry-run mode so I could test without touching real billing data, which made every subsequent change safe. I deliberately did not propose a rewrite, because the 200 lines of edge-case handling that looked like mess turned out to encode 3 years of real customer situations. After 2 months I understood it well enough to refactor the slow phase, which took it to a consistent 25 minutes.

What gets you hired

Our environment provisioning took about 45 minutes of manual clicking and 2 engineers did it maybe 8 times a month, so roughly 12 hours monthly, and about 1 in 5 came out subtly different, which caused debugging later that never got attributed back to the provisioning. So the real cost was higher than the 12 hours. Before automating I asked whether it needed doing at all, and part of it did not: 2 of the steps existed for a system we had decommissioned. Removing those was the cheapest win and took an hour. Then I automated the rest in Terraform, which took about 3 days including making it reliable enough to trust. The payback on pure time was under a month, but honestly the bigger wins were consistency, since every environment now matches, and removing the single point of knowledge, because previously only 2 of us could do it. Afterwards I checked the actual numbers rather than assuming, and provisioning went to about 6 minutes of waiting with no human steps.

What gets you hired

A managed database was showing replication lag climbing during an incident and I had never dealt with that specific failure in that service. What I did not know was whether promoting the replica was safe or would lose data. Rather than guessing on a production database, I bounded the risk first: I took a manual snapshot, which took 2 minutes and made the worst case recoverable, and I stopped the write path feeding the lag so it could not get worse while I thought. Then I closed the gap fast in parallel: I read the vendor documentation on promotion semantics and messaged a colleague who had run this database at a previous company. Within about 10 minutes I knew promotion would be safe with a bounded loss window. I said explicitly in the channel that I was 80% confident, not certain, and what my rollback was, because I would rather be visibly uncertain than quietly wrong. It worked. What I learned beyond the technical detail was that stating a confidence level made everyone else more useful, since 2 people immediately offered relevant experience.

What gets you hired

We were asked to ship a payments change for a launch 3 days out, and the load testing was not done. Rather than saying no, I asked what the date was tied to, which turned out to be a marketing campaign already booked, so the date was genuinely hard. Then I made the risk specific: without load testing, my honest estimate was a meaningful chance of degradation at campaign traffic, which is 5 times normal, and the failure mode is failed payments, not a slow page. That is a number a business person can weigh. Then I brought 3 options: ship the full scope on the date and accept the risk, ship behind a flag enabled for 10% of traffic and ramp during the campaign, or cut one non-essential part and load test the rest. They chose the flagged ramp, which I thought was the right call. I wrote the decision and reasoning in the channel so it was shared rather than mine. It shipped, we caught a connection-pool limit at 10% and fixed it before full ramp.

What gets you hired

I wanted a team to move from hand-written Kubernetes manifests to a shared template, and they pushed back hard. Instead of arguing, I asked why, and the real reason was that a previous platform change had broken their deploys during a launch and nobody helped them fix it. That was a trust problem, not a technical one. So I made the first step tiny: I offered to migrate 1 non-critical service myself, with them reviewing, and committed that if anything broke I would be the one to fix it at any hour. That took 2 days. The template caught a missing resource limit they had not noticed, which was concrete value on their own service rather than my slide. They migrated the next 3 themselves over a month. I also changed the template based on their feedback, since they needed an override the original design refused, and that override turned out to be right for other teams too. The lesson was that the resistance contained real information.

What gets you hired

Security required a manual review before any production deploy, which would have taken us from about 10 deploys a day to 2 a week. Instead of arguing about speed, I asked what risk the review was actually catching, and the honest answer was unreviewed code and unscanned dependencies reaching production. Both are legitimate. So I proposed meeting the same intent automatically: required pull request review enforced by branch protection so nothing self-merges, dependency and image scanning as a hard gate, and a signed evidence trail per deploy recording commit, approver, scan results and artifact digest. Their concern was that they could not verify it, so I built them a dashboard showing every production deploy with its evidence, and we agreed a quarterly sample audit. That gave them stronger assurance than a human eyeballing a diff at 5pm, because it is enforced rather than attested. The bigger change was that I started bringing them in at design time, so we now write controls together instead of negotiating after I have built something.

What gets you hired

We had a deployment system with 3 abstraction layers built over 4 years: a wrapper around Helm, a wrapper around that wrapper, and a set of per-team overrides. It was 6000 lines and every new engineer needed about 2 weeks to make their first deploy safely. Each layer had solved a real problem at the time, which is why nobody had touched it. I started with evidence rather than opinion: I instrumented which override paths were actually used and found that 4 of 19 accounted for nearly all usage, and 9 had not been used in over a year. So I built a thin replacement supporting the 4, migrated 2 friendly services, and ran both for 6 weeks. Then I removed the unused paths in stages, disabling before deleting so anything I got wrong was a config flip to restore. The result was about 800 lines instead of 6000, onboarding to first deploy went from 2 weeks to 2 days, and pipeline duration dropped by roughly 4 minutes. Nobody has asked for the removed features.

What gets you hired

A release was ready on a Friday afternoon and it included a change to how we wrote to the payments ledger. My concern was specific: the change altered write ordering, and I could not see a test covering the partial-failure case where the first write succeeded and the second did not. Before raising it I checked, because blocking on a false alarm costs credibility I would need later, and I confirmed there was genuinely no coverage for that path. I raised it directly with the engineer and the product owner together rather than commenting in a pull request, and I framed it as the consequence, that a partial write would leave a customer charged with no record, rather than as a coverage percentage. Then I made it a condition rather than a veto: I said I would be comfortable shipping Monday with a test for that path, or shipping now behind a flag disabled for real customers. They chose Monday. The test they wrote failed on the first run, which found the bug. It cost 3 days and I would do it again.

What gets you hired

I started with numbers rather than the general feeling that on-call was bad. Over 8 weeks we averaged about 11 pages a week, 6 of them between midnight and 6am, and when I categorised them, roughly 60% came from 3 recurring causes and about half were not actionable at all. That reframed it entirely: it was not a rota problem, it was an alerting and reliability problem. So first I deleted the non-actionable alerts, which took nightly pages down immediately and cost nothing. Then the 3 recurring causes: one was a disk filling weekly, fixed with rotation and an earlier warning threshold, one was a dependency without a timeout causing pile-ups, and one was a known memory leak we had been restarting manually. Fixing those 3 removed most of the remaining volume. Only then did I touch the rota, adding a proper handover and time back after a bad night. Pages went from about 11 a week to 2, and the important part is that people stopped dreading the pager, which showed up in how quickly they responded.

What gets you hired

I proposed and got approval for a service mesh, arguing for mTLS, traffic shifting and better observability. We had spent about 6 weeks on it with 2 engineers when I started tracking whether the benefits were landing. They were not, for us specifically: we had 12 services with modest traffic, our tracing gap was really a missing instrumentation problem the mesh did not solve, and the sidecar added latency and a genuinely hard debugging layer for a team with no prior experience of it. Meanwhile 2 of our 4 recent incidents were mesh-related, in a system it was supposed to make more reliable. I brought that to the team and argued against my own proposal, which was uncomfortable because it was publicly mine and 6 weeks were spent. We stopped and instead did direct OpenTelemetry instrumentation and cert management at the ingress, which got us most of what we wanted in about 3 weeks. The reason I did it that way, publicly and with data rather than quietly letting it fade, was that if I cannot reverse my own decision on evidence, nobody junior to me will feel able to reverse theirs.

What gets you hired

First identify the consumer with top sorted by CPU, which takes 5 seconds. But the more informative thing is the breakdown across user, system, iowait and steal, because each points somewhere different. High user time is the application genuinely computing. High system time means excessive syscalls or kernel work, often a context switching storm or something hammering the network stack. High iowait means the CPU is idle waiting for disk, so the real problem is storage rather than compute, and adding CPU would change nothing. High steal on a VM means the hypervisor is giving my cycles to a noisy neighbour, which is not my problem to fix locally. I would also read load average with that in mind, since Linux load counts uninterruptible processes too, so load of 40 with 10% CPU is an IO problem wearing a CPU costume. Then narrow: per-thread view to find the hot thread, then a profiler or strace to see what it is actually doing. Only then decide, and I would not kill a user-serving process without understanding it first.

What gets you hired

SIGTERM is a polite request the process can catch, so it can finish what it is doing. SIGKILL cannot be caught and the kernel terminates immediately. In Kubernetes the sequence is: the pod is removed from the service endpoints, SIGTERM is sent, and after the termination grace period, 30 seconds by default, SIGKILL follows. What the app does in that window is the difference between a clean deploy and dropped requests: it should stop accepting new connections, finish in-flight requests, close database connections cleanly, and exit. If it exits instantly on SIGTERM, every in-flight request is cut. The trap that catches people is PID 1: a process running as PID 1 in a container does not get the kernel default handlers, so if it has no explicit SIGTERM handler the signal does nothing and every pod takes the full 30 seconds and then dies hard. And if the entrypoint is a shell script, the shell often does not forward the signal to the child at all, so the app never even sees it. Using exec in the entrypoint or a minimal init process fixes that.

What gets you hired

I would start with constraints: are any of the 15 coupled such that they must deploy together, since that changes everything. Assuming mostly independent, the design is a shared pipeline template owned by a platform team, consumed by all 15 services, so a change like adding image signing lands everywhere in 1 pull request rather than 15. Each service builds once, tags by commit sha, and promotes that identical artifact through environments, with a GitOps agent reconciling per environment so deploys are commits rather than jobs holding cluster credentials. Independent deployability then depends on contract compatibility, so I would require backward-compatible API and schema changes and use consumer-driven contract tests to catch breakage before merge, because that is what actually lets 4 teams ship without a release train. Guardrails go in the template as automated policy, so the security team writes rules once instead of reviewing 40 deploys a week. And I would track DORA metrics per service to see which team is struggling rather than guessing.

What gets you hired

The stated failure is that nobody can explain a slow request, which is a tracing gap, so tracing is where I would start rather than more dashboards. I want every request to carry a trace ID from the edge, propagated across service boundaries, so a single slow request shows its span breakdown and I can see that 400ms of a 500ms request was one database call. Then metrics for detection, alerting on user-visible symptoms tied to an SLO, since traces are for investigating and metrics are for noticing. Then structured logs that include the trace ID, which is the piece teams usually miss: without it you have three systems and a manual correlation problem, and with it you go alert to trace to logs in about 3 clicks. I would standardise instrumentation through OpenTelemetry so the vendor choice stays reversible. Cost control matters: head sampling at a few percent with tail sampling to always keep errors and slow requests, plus retention tiers, because full-fidelity tracing at scale gets expensive quickly.

What gets you hired

The hard part is not storage, it is the bootstrap: how does a service prove it is allowed to read anything. I would use platform workload identity, so a pod authenticates as its own service account and the secret store authorises that identity directly, which means there is no long-lived token anywhere and no chicken-and-egg problem. Policies are scoped per service and environment, so the payments service in staging can read only its own staging secrets, and a compromise of one service does not expose the other 39. Where the backend supports it I prefer dynamic credentials: the store generates a database user on demand with a 1 hour lease, so rotation becomes continuous rather than a scary quarterly event. For audit, every read is logged with identity, secret path and timestamp, shipped to a store the platform team cannot silently edit, which is what an auditor actually needs. Then the operational realities: caching so the store is not in the hot path of every request, and a documented break-glass procedure that is heavily alerted rather than pretending emergencies never happen.

What gets you hired

The design lever is what is duplicated versus shared. I would give each PR its own namespace with the application services, and share the expensive stateful pieces where safe. Data is the deciding question: a full production-sized database per PR is unaffordable and slow, so I would use a small seeded dataset created from a template, ideally a snapshot restore or a copy-on-write clone that takes 30 seconds rather than a migration run that takes 10 minutes. Lifecycle fully automated: created on PR open, updated on each push, destroyed on merge or close, plus a hard TTL of about 3 days so abandoned branches cannot accumulate, since with 30 engineers the forgotten environments are what actually generates the bill. Cost control means scaling to zero when idle, because most preview environments are used for 20 minutes and exist for 2 days. Then usefulness: post the URL as a PR comment automatically, seed realistic data, and give it a way to send test traffic, otherwise it becomes an expensive thing nobody opens.

What gets you hired

I would start by pinning the numbers, because 15 minutes recovery time with zero data loss and 15 minutes with 5 minutes of acceptable loss are completely different systems and costs. Then the data layer, since stateless compute is the easy part. For a 15 minute target, active-passive is almost certainly right: a warm standby region with asynchronous database replication, which means an RPO equal to replication lag, typically seconds. Active-active with synchronous writes buys a lower RTO and costs cross-region write latency on every request plus conflict resolution, and I would need a strong reason. So: infrastructure defined once as code and deployed to both regions, compute running warm at reduced capacity, database replicating asynchronously, and object storage replicated. Traffic steering by health-checked DNS, accepting that client caching adds minutes, which is why 15 rather than 2. The critical piece people skip is promotion: promoting a replica is the risky step, and it needs to be automated and rehearsed, with fencing so the old primary cannot accept writes and cause split brain. Then quarterly game days, because an untested failover is a hope.

What gets you hired

My job as commander is coordination, not debugging, and the first mistake to avoid is joining the investigation myself. I state roles out loud immediately: I am commanding, one person owns communications, and I assign specific investigation areas so 6 people are not all reading the same dashboard. Then mitigation before diagnosis: is there a recent deploy to roll back, can we fail over, can we shed traffic to restore partial service. Understanding why can wait until users are served. I enforce one change at a time, announced in the channel before it happens, because 3 simultaneous fixes make it impossible to know what helped or hurt. I keep a timeline as we go, with timestamps, which coordinates the room now and writes most of the postmortem later. I communicate on a fixed cadence, every 30 minutes even with nothing new, because silence makes stakeholders escalate and interrupt the responders. And at 2am I plan the handover early, since tired engineers cause the second incident.

What gets you hired

The key number is how long capacity actually takes to arrive, end to end. That is metric scrape delay, plus the controller decision interval, plus scheduling, plus possibly a node coming up, plus image pull, plus application warm-up. Realistically that is 2 to 5 minutes, and if traffic triples in 60 seconds, no reactive system can win: it is arithmetic, not tuning. So for a known 9am spike I would use scheduled scaling to pre-warm capacity at 8:45, because the pattern is predictable and pre-warming is far cheaper than the outage. Reactive autoscaling stays on top of that for the unpredictable component. I would also choose the signal carefully: CPU is a poor proxy for an IO-bound service, and requests per replica or queue depth track demand much more directly. Then defence in depth for the case where I am wrong: a queue or load shedding so excess requests degrade gracefully rather than collapsing every request, and keeping some headroom rather than running at 90% utilisation, since utilisation and latency are not linearly related near saturation.

What gets you hired

My default is managed, and the reason is not laziness, it is that the managed service is buying an operational function rather than a server. Automated backups with point-in-time recovery, patching, failover, and someone else awake at 3am when a disk fails. Self-managing means my team owns all of that, including the part everyone underestimates, which is being able to actually restore under pressure, because an untested backup is a hope. So the price comparison people make is wrong: it is not instance cost versus managed cost, it is instance cost plus roughly a portion of an engineer permanently. That said, there are real exceptions. An extension or a version the managed service does not support, which is common with Postgres extensions. Performance requirements needing filesystem or kernel tuning the service does not expose. Regulatory constraints requiring specific residency or control. Or scale where the managed premium becomes millions rather than thousands, at which point hiring dedicated database engineers is genuinely cheaper. I would ask for one of those specific reasons, because "we can run it ourselves" is true and not sufficient.

What gets you hired

I would resist announcing a standard in my first month, because the interesting question is why 200 engineers each built their own. Usually the answer is that the central path did not exist or was slower than doing it themselves, and if I do not fix that cause, my new standard becomes the eleventh tool. So first, discovery: what do teams actually run, where do they lose time, what does deploying feel like. Then find the common 80%, which is usually build, test, artifact, deploy, rollback, and standardise exactly that while leaving legitimate differences alone. Then a paved road that is genuinely faster than rolling your own, proven with 2 or 3 volunteer teams and measured, so I can say lead time went from 4 days to 1 rather than arguing about elegance. That result recruits the next teams. Only once adoption is voluntary and real do I set a decommission date, and I resource the migration rather than sending a deadline email. I would expect this to take 4 to 6 quarters, and I would say that up front rather than promise a quarter and lose credibility.

What gets you hired

I would advise against it more often than people expect. Kubernetes is a permanent operational commitment: cluster upgrades, networking, RBAC, admission policy, and enough shared understanding that the whole team can debug it at 3am. That cost is worth paying when you have many services, real elasticity requirements, and several teams needing self-service isolation. A company with 3 services, predictable traffic, and 6 engineers gets almost none of that return and pays the full price, and typically one engineer quietly becomes the cluster person and stops shipping product. For them a managed container service or an application platform gives containers, autoscaling, and rolling deploys with a tiny fraction of the surface. The reason I am comfortable saying this is that the durable investment is containerising and having clean infrastructure as code, and that work transfers. Moving to Kubernetes in 2 years when the shape of the problem justifies it is a manageable migration. Adopting it early and drowning in it is not easily undone.

What gets you hired

I would refuse the framing, because the data on this is consistent: teams that deploy more often also restore faster and fail less, since small batches and good automation produce both. So the real question is what specifically is causing incidents. I would get numbers first: change failure rate, incident causes grouped, and how on-call time actually splits between real incidents and noise. Usually 2 causes produce most of the pain, and one of them is alert noise rather than genuine outages. Then I would propose an error budget: agree a target with the business, say 99.9%, and agree in advance that if we exceed the budget, work shifts to reliability until it recovers. That converts a recurring argument into a rule everyone signed, and crucially it also means that while we are inside budget the business gets to ship fast without me objecting, which is what makes it acceptable to them. Meanwhile I fix on-call immediately: delete non-actionable pages, size the rota properly, and give recovery time, because a tired team causes the next incident.

What gets you hired

The first question is whether this differentiates us. Customers do not choose us because of our deploy tooling, so it is a cost centre, and the default for a cost centre is buy. Then I cost the build honestly, which people rarely do: it is not the 6 months to version one, it is 2 engineers permanently maintaining it, being on call for it, and the product work those engineers did not do, which over 3 years usually exceeds a vendor bill by a wide margin. I would also name the bias out loud, that engineers including me enjoy building platforms and systematically underestimate maintenance, so the estimate deserves scepticism. Against that, I check lock-in: how hard is exit, and can we keep an open standard underneath so migration stays possible. Usually the honest answer is hybrid: buy the commodity layers, build the thin integration that encodes our specific context, because that is where our knowledge actually matters and it is small enough to maintain. I would trial the vendor on 2 teams with defined success criteria before committing.