Back to Docker: Images, Containers, Multi-Stage
Q6Question 6 of 10

Scanning Docker Images for Vulnerabilities and Managing Secrets Inside Containers Securely

Key Concepts
  • Scan images with Trivy or Grype right after build, failing the pipeline on high or critical CVEs.
  • Registry scanning re-scans stored images as new CVEs are published, catching vulnerabilities discovered after the initial scan.
  • Never bake secrets into image layers via ENV or COPY because layers are readable and persistent.
  • Inject secrets at runtime as mounted Kubernetes Secrets or via the External Secrets Operator syncing from Vault.
  • Prefer mounted files over environment variables because env vars leak more easily into logs and crash dumps.
  • Use BuildKit secret mounts for build-time secrets so they are never persisted in any image layer.

Definition

Scan images for known CVEs with Trivy or Grype and block severe findings before deploy, and never bake secrets into layers; inject them at runtime from a secret manager.
  • Image scanning – analysing image layers, OS packages, and application dependencies against CVE databases before deployment.
  • Trivy/Grype – vulnerability scanners that report findings by severity and fail the build on high or critical issues.
  • Registry scanning – continuous scanning of stored images as new CVEs are published.
  • Never in layers – no ENV DB_PASSWORD=... or secrets files copied into the image, because layers are readable and persistent.
  • Runtime injection – secrets provided at container start as mounted files or environment variables from Kubernetes Secrets or a secret manager.
  • BuildKit secret mounts – RUN --mount=type=secret for build-time secrets that are not persisted in any layer.

Scanning an image is like the National Bank of India inspecting every component of a new cash-counting machine against a recall list before installing it. If any part is recalled (known CVE), the machine does not get installed. Managing secrets is like the vault combination: you never stamp it onto the machine's casing (image layer). The branch manager receives it fresh each morning in a sealed envelope that is used and destroyed (runtime-mounted secret).

Understanding the Concepts

How does image scanning work?

A scanner inspects the image's OS packages and application dependencies, comparing each against CVE databases. It reports findings by severity.

BASH
# Trivy: scan an image right after build
trivy image --exit-code 1 --severity HIGH,CRITICAL \
  registry.bank.local/transfer-service:abc1234

# Example output:
registry.bank.local/transfer-service:abc1234 (debian 12.4)

Total: 3 (HIGH: 2, CRITICAL: 1)

+-----------------+----------------+----------+-------------------+
| Library         | Vulnerability  | Severity | Fixed Version     |
+-----------------+----------------+----------+-------------------+
| log4j-core      | CVE-2021-44228 | CRITICAL | 2.17.0            |
| jackson-databind| CVE-2022-42003 | HIGH     | 2.14.0            |
| libssl3         | CVE-2023-5678  | HIGH     | 3.0.13-1          |
+-----------------+----------------+----------+-------------------+
# Exit code 1: build fails, image cannot be promoted.
BASH
# Grype: alternative scanner
grype registry.bank.local/transfer-service:abc1234 --fail-on high

When does scanning happen in the pipeline?

Scanning happens at multiple points for defence in depth.

YAML
# Pipeline: scan right after image build
jobs:
  build-and-scan:
    steps:
      - run: mvn clean package
      - run: docker build -t $IMAGE .
      - run: trivy image --exit-code 1 --severity HIGH,CRITICAL $IMAGE
      - run: docker push $IMAGE   # Only reached if scan passes
TEXT
  Scanning points:
  1. Pipeline (after build)    – block before promotion
  2. Registry (continuous)     – catch new CVEs on stored images
  3. Admission control (K8s)   – block unscanned images at deploy time

What is registry scanning?

Continuous scanning re-scans stored images as new CVEs are published. An image that was clean last week may be flagged today.

BASH
# Harbor registry: automatic scanning on push and on schedule
# Alert when a stored image becomes vulnerable to a new CVE.
# No pipeline change needed; the registry does it.

Why must secrets never be in image layers?

Every Docker image layer is readable by anyone who can pull the image. A secret in a layer persists forever, even if a later instruction deletes the file.

DOCKERFILE
# WRONG: ENV bakes the password into the image config.
FROM eclipse-temurin:21-jre
ENV DB_PASSWORD=S3cur3P@ssw0rd!
COPY target/app.jar /app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]
# docker inspect reveals DB_PASSWORD to anyone.

# WRONG: COPY then delete. The file is still in the COPY layer.
COPY secrets.properties /app/
RUN rm /app/secrets.properties
# docker save | tar xf reveals secrets.properties.

# CORRECT: no secrets in any layer.
FROM eclipse-temurin:21-jre
COPY target/app.jar /app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]
# Secrets come from Kubernetes Secrets at container start.

How do containers receive secrets at runtime?

Secrets are mounted from Kubernetes Secrets or synced from an external manager.

YAML
# Kubernetes: mount secret as a file
apiVersion: v1
kind: Secret
metadata:
  name: transfer-secrets
data:
  DB_PASSWORD: UzNjdXIzUEBzc3cwcmQh   # base64
---
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: transfer-service
        volumeMounts:
        - name: secrets
          mountPath: /app/secrets
          readOnly: true
      volumes:
      - name: secrets
        secret:
          secretName: transfer-secrets
YAML
# External Secrets Operator: sync from Vault
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: transfer-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: transfer-secrets
  data:
  - secretKey: DB_PASSWORD
    remoteRef:
      key: secret/data/banking/transfer
      property: db_password

What are BuildKit secret mounts?

For build-time secrets (like a private registry token), use RUN --mount=type=secret so the secret is available during the RUN step but never persisted in any layer.

DOCKERFILE
# WRONG: build arg leaks into image history.
ARG REGISTRY_TOKEN
RUN echo "token=${REGISTRY_TOKEN}" > /app/.npmrc

# CORRECT: BuildKit secret mount (not in any layer).
RUN --mount=type=secret,id=registry_token \
    REGISTRY_TOKEN=$(cat /run/secrets/registry_token) && \
    echo "token=${REGISTRY_TOKEN}" > /tmp/.npmrc && \
    npm install && rm /tmp/.npmrc
BASH
# Build with BuildKit secret
DOCKER_BUILDKIT=1 docker build \
  --secret id=registry_token,src=./token.txt \
  -t $IMAGE .

How do you manage findings?

Not every finding is fixable immediately. Triage by severity and exploitability, fix by upgrading, and record justified suppressions with expiry dates.

BASH
# Fix: upgrade the vulnerable library
mvn versions:use-latest-versions -Dincludes=com.fasterxml.jackson

# Suppress with justification (Trivy)
# .trivyignore
CVE-2023-9999  # Accepted risk: not exploitable in our context. Expires 2025-03-01.

What should a senior engineer know?

  • Defence in depth: pipeline scanning + registry scanning + admission control + image signing + SBOM.
  • Mounted files are preferred over environment variables because env vars leak into logs and crash dumps.
  • Short-lived, dynamically issued credentials from Vault auto-expire, so even a leaked value is quickly useless.
  • Per-environment scoping: a leaked QA credential cannot touch Prod.
  • Every access to the secret store is audited so you can answer who read the production password and when.
YAML
# Kubernetes: admission control blocks unscanned images
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
# Configured to reject pods with images not signed or scanned.

Visual Overview

DIAGRAM
  Image scanning flow:

  docker build ──▶ trivy image ──▶ Pass? ──▶ docker push ──▶ Registry
                        │                                      │
                   Fail (HIGH/CRIT)                     Continuous scan
                        │                               (new CVEs found)
                   Block promotion                            │
                                                         Alert team

  Secrets flow:

  ┌──────────────────┐
  │  Secret Store     │  (Vault / AWS SM)
  └────────┬─────────┘
           │ sync
           ▼
  ┌──────────────────┐
  │  K8s Secret       │  (External Secrets Operator)
  └────────┬─────────┘
           │ mount
           ▼
  ┌──────────────────┐
  │  Container        │  reads /app/secrets/DB_PASSWORD
  │  (no secret in    │
  │   any layer)      │
  └──────────────────┘

Key Takeaway

Scanning Docker images means analysing their layers, OS packages, and dependencies against CVE databases before deployment. I run Trivy or Grype as a pipeline step right after the image is built, with the scanner set to fail the build on high or critical findings. Registry scanning re-scans stored images as new CVEs are published, catching vulnerabilities discovered after the initial build. In Kubernetes, admission control blocks images that have not been scanned or signed.

Managing secrets inside containers starts with a hard rule: secrets never go into the image. No ENV with a password, no secrets file copied in, because every layer is readable and persistent. The image contains only code. Secrets are provided at runtime from Kubernetes Secrets synced from a dedicated manager like Vault, preferably as mounted files rather than environment variables because env vars leak more easily into logs and crash dumps.

For build-time secrets, BuildKit secret mounts ensure the secret is available during the RUN step but never persisted in any layer. For the strongest posture, use short-lived dynamically issued credentials that auto-expire, per-environment scoping, and full audit logging.

Remember one sentence: scan images for CVEs before they ship, and inject secrets at runtime from a store so the image itself never contains a sensitive value.

---

Share
Use ← → arrow keys to navigate