Container & Image Security Scanning

The CVE you shipped without writing a line of code

Friday afternoon. Your service is healthy, your tests are green, your code review was spotless. On Monday, security pages you: your production image contains a critical remote-code-execution CVE. You didn't write the vulnerable code, it came in your base image, three layers down, in an OpenSSL build you never chose and never knew was there.

This is the uncomfortable truth of containers: when you write FROM node:20, you inherit an entire operating system, hundreds of OS packages, each with its own vulnerability history. Add your npm install and you pull in a tree of transitive dependencies that no human has read. The average image ships with dozens of known CVEs before your code even runs. The fix isn't heroics at 2 a.m. It's moving the check to the left, scanning every image, on every build, and refusing to ship the broken ones.

Shift left: find it where it's cheap to fix

A vulnerability caught in CI costs minutes to fix. The same vulnerability caught in production costs an incident, a postmortem, and your weekend.

"Shift left" means moving security checks earlier in the timeline, to the *left* of the pipeline, toward the developer, instead of bolting them on at the end. A scanner is just a tool that reads what's inside an image (its packages and their versions), looks each one up in public vulnerability databases, and tells you which ones have known CVEs. The leverage isn't the scan; it's *where* you run it and *whether it can stop the line*.

The picture: build, gate, deploy

Here is the full path an image travels from your laptop to a running pod. The two control points that matter are the gate in CI (fail on critical) and admission control at the cluster (reject anything that didn't go through the gate).

1. 1

   Build the image

   Your CI runs `docker build` and produces a tagged image, base layers plus your app.

2. 2

   Scan the image

   Trivy or Grype reads every layer, enumerates packages, and matches versions against CVE feeds.

3. 3

   Gate on severity

   The scanner exits non-zero if it finds a CRITICAL (or HIGH). A non-zero exit fails the CI job, the line stops.

4. 4

   Push only if clean

   A passing build pushes the image to your registry. A failing build never gets that far.

5. 5

   Enforce at admission

   When something tries to deploy, an admission controller checks the image was scanned/signed and rejects it otherwise.

What to scan (it's more than OS packages)

A common mistake is thinking "image scanning" means "OS package scanning." Your attack surface is wider: the application dependencies you bundle, the infrastructure-as-code that provisions everything, and any secrets that accidentally got baked into a layer. Good news, modern scanners like Trivy cover all four from one binary.

The gate: a CI step that fails on HIGH/CRITICAL

This is the whole point of the article. Scanning that only *reports* is theatre, someone has to read the report, and nobody does. The gate makes the pipeline read it for you. Here's a GitHub Actions job that builds the image, scans it with Trivy, and fails the build the moment a HIGH or CRITICAL with a known fix appears.

name: build-and-scan

on:
  pull_request:
  push:
    branches: [main]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build image
        run: docker build -t app:${{ github.sha }} .

      - name: Scan image with Trivy
        uses: aquasecurity/trivy-action@0.24.0
        with:
          image-ref: app:${{ github.sha }}
          format: table
          # Fail the job on these severities...
          severity: HIGH,CRITICAL
          # ...but only when an upstream fix actually exists.
          ignore-unfixed: true
          # exit-code: 1 turns a finding into a failed build (the gate).
          exit-code: "1"

      - name: Push image
        # Only runs if the scan step above passed.
        if: github.ref == 'refs/heads/main'
        run: |
          echo "Scan clean, pushing image"
          docker push app:${{ github.sha }}

• `exit-code: "1"` is the line that turns a scan into a *gate*. Without it, Trivy prints findings and exits 0, the build stays green and nothing stops.
• `ignore-unfixed: true` avoids failing on CVEs that have no patch yet, you can't fix what upstream hasn't fixed, so failing there just trains people to ignore the gate.
• `severity: HIGH,CRITICAL` is your threshold. Start strict on CRITICAL, add HIGH once your images are clean, and resist the urge to also fail on LOW (noise kills trust in the gate).
• The push step is guarded by if: so a failed scan never reaches the registry.

Shrink the target: distroless & minimal bases

The cheapest vulnerability to fix is the one that was never in the image. Every package in your base is a package that can have a CVE. A full ubuntu or node image carries a shell, a package manager, and hundreds of libraries your app never calls, pure attack surface. Distroless and minimal base images strip that down to just your app and its runtime.

Pair this with a multi-stage build: compile in a fat builder stage, then copy only the artifact into a distroless final stage. You get a small, fast image with almost nothing for a scanner to flag.

Generate an SBOM while you're at it

A Software Bill of Materials (SBOM) is a machine-readable inventory of everything in your image. Trivy can emit one (trivy image --format cyclonedx). The win: when the *next* big CVE drops, you don't rebuild and rescan every image to find out who's affected, you query your stored SBOMs. Scanning tells you what's vulnerable *today*; the SBOM lets you answer that question instantly *tomorrow*. This is the heart of supply-chain security, know exactly what you ship.

One more gate: runtime & admission control

CI scanning assumes everything reaches production *through* CI. In practice, someone will kubectl apply a hand-tagged image, or a stale one will get redeployed. Admission control is the cluster's own gate: a controller (Kyverno, OPA Gatekeeper, or a registry policy) inspects every image at deploy time and rejects ones that weren't scanned, weren't signed, or pull from an untrusted registry. CI is the lock on the front door; admission control is the lock on the back door, you want both.

• Require signed images, only admit images signed by your CI (e.g. with cosign), so a random :latest from Docker Hub can't deploy.
• Restrict registries, allow pulls only from your own registry, where every image has already passed the gate.
• Block privileged pods, reject containers asking for root, host networking, or privileged: true at admission.
• Runtime monitoring, even an admitted image can be exploited later; tools like Falco watch for suspicious behavior in running containers.

Common mistakes that cost hours

1. Scanning but never gating. Trivy in a step with exit-code: 0 produces a beautiful report nobody reads. If a finding can't fail the build, it doesn't exist.
2. Fat base images. FROM ubuntu then chasing 80 CVEs every week. Move to distroless/alpine and most of them simply vanish, you can't have a CVE in a package you didn't install.
3. Ignoring transitive dependencies. You audited your direct package.json, but the CVE is four levels deep in a dependency-of-a-dependency. Scanners walk the *whole* tree; lockfiles are where the real risk lives.
4. Failing on unfixable CVEs. Gating on findings with no available patch just teaches everyone to slap || true on the scan. Use ignore-unfixed and a dated .trivyignore.
5. Scanning only at release. A scan once a quarter means you find out about Friday's CVE in three months. Scan every build, and re-scan on a schedule since new CVEs land against images you already shipped.

Takeaways

Where to go next

Scanning is one pillar of secure delivery. Pair it with images that are small by design and a supply chain you can trust end to end.

• Build images that barely register on a scanner: Dockerfile Best Practices: Small, Fast, Secure Images.
• Go end-to-end on provenance and signing: Securing the Software Supply Chain.
• Practice the build → scan → gate loop hands-on in the Docker lab.
• See where this fits in the bigger picture: the DevOps Engineer career path.