Definition
- Never in Git – no passwords in
application.yml, no cloud keys in.env, no JWT secrets in properties files. - Never in images – no
ENV DB_PASSWORD=...or secrets copied into a Dockerfile layer, because every layer is readable. - Secret store – a dedicated manager (HashiCorp Vault, AWS Secrets Manager, or the CI system's encrypted store) holds the real values.
- Masked injection – the pipeline injects secrets as environment variables only for the step that needs them, automatically masking them in logs.
- Short-lived credentials – prefer OIDC federation or dynamically issued credentials that expire in minutes over static keys.
- Secret scanning – a pre-commit scanner like gitleaks fails the build if a secret is ever committed.
Think of the National Bank of India's vault keys. You would never write the combination on the noticeboard (source code), print it on every delivered parcel (image layer), or read it aloud over the intercom (build logs). Instead the combination lives in a central key office (the secret store). When a teller needs it, the office hands over a temporary code that expires in minutes (short-lived credentials), and the office keeps a log of who asked and when (audit).
Understanding the Concepts
Why must secrets never live in Git?
Once a secret is committed to Git, it is in the history forever. Even if you remove it in a later commit, anyone with clone access can find it. A pre-commit scanner catches this before it happens.
# WRONG: password committed in application.yml
spring:
datasource:
url: jdbc:postgresql://db.bank.local:5432/transfers
username: transfer_svc
password: S3cur3P@ssw0rd! # Committed to Git!
# CORRECT: placeholder resolved at runtime
spring:
datasource:
url: jdbc:postgresql://db.bank.local:5432/transfers
username: transfer_svc
password: ${DB_PASSWORD} # Resolved from environment at runtime
# Wire gitleaks into CI to fail the build on committed secrets
gitleaks detect --source . --verbose
# Exit code 1 if secrets found, 0 if clean.
Why must secrets never be baked into images?
Every Docker image layer is readable by anyone who can pull the image. A secret in a layer persists in the registry forever, even if a later layer deletes the file.
# WRONG: secret baked into the image as an ENV
FROM eclipse-temurin:21-jre
ENV DB_PASSWORD=S3cur3P@ssw0rd!
COPY target/app.jar /app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]
# Anyone who pulls this image can read DB_PASSWORD.
# WRONG: secret file copied into the image
FROM eclipse-temurin:21-jre
COPY secrets.properties /app/secrets.properties
# Even if a later RUN rm deletes it, it is still in the earlier layer.
# CORRECT: image contains only code, secrets injected at runtime
FROM eclipse-temurin:21-jre
COPY target/app.jar /app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]
# DB_PASSWORD comes from a Kubernetes Secret at container start.
Where do secrets come from?
Secrets come from a dedicated store. The pipeline reads from the store at the moment of use.
# GitHub Actions: encrypted secrets injected as masked variables
jobs:
deploy:
steps:
- name: Push image
env:
REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
run: |
echo "$REGISTRY_PASSWORD" | docker login registry.bank.local -u ci-bot --password-stdin
docker push registry.bank.local/transfer-service:$SHA
# REGISTRY_PASSWORD is masked in logs as ***
// Jenkins: credentials from the credential store
pipeline {
agent any
stages {
stage('Deploy') {
steps {
withCredentials([
string(credentialsId: 'db-password', variable: 'DB_PASSWORD'),
string(credentialsId: 'k8s-token', variable: 'KUBE_TOKEN')
]) {
sh "kubectl --token=${KUBE_TOKEN} create secret generic transfer-secrets " +
"--from-literal=DB_PASSWORD=${DB_PASSWORD} -n prod"
}
}
}
}
}
How do Kubernetes workloads receive secrets?
The running application reads secrets from a mounted Kubernetes Secret or from an external secret synced by the External Secrets Operator.
# Kubernetes Secret mounted as a file
apiVersion: v1
kind: Secret
metadata:
name: transfer-secrets
namespace: prod
type: Opaque
data:
DB_PASSWORD: UzNjdXIzUEBzc3cwcmQh # base64 encoded
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: transfer-service
spec:
template:
spec:
containers:
- name: transfer-service
image: registry.bank.local/transfer-service:abc1234
volumeMounts:
- name: secrets
mountPath: /app/secrets
readOnly: true
volumes:
- name: secrets
secret:
secretName: transfer-secrets
# External Secrets Operator: syncs from Vault
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: transfer-secrets
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: transfer-secrets
data:
- secretKey: DB_PASSWORD
remoteRef:
key: secret/data/banking/transfer
property: db_password
What are short-lived credentials and OIDC?
The strongest pattern is no static secret at all. The CI job authenticates using OpenID Connect federation, exchanging a signed identity token for temporary cloud credentials.
# GitHub Actions: OIDC to AWS (no stored access key)
jobs:
deploy:
permissions:
id-token: write
steps:
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789:role/ci-deploy
aws-region: ap-south-1
# No AWS_ACCESS_KEY_ID stored anywhere.
# GitHub exchanges a signed JWT for temporary STS credentials.
- run: aws ecr get-login-password | docker login ...
What about BuildKit secret mounts?
A secret passed as a --build-arg can end up in image history. BuildKit secret mounts avoid this.
# WRONG: build arg leaks into image history
ARG NPM_TOKEN
RUN echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > .npmrc
# CORRECT: BuildKit secret mount (not persisted in any layer)
RUN --mount=type=secret,id=npm_token \
NPM_TOKEN=$(cat /run/secrets/npm_token) \
echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > .npmrc && \
npm install && \
rm .npmrc
# Build with BuildKit secret
DOCKER_BUILDKIT=1 docker build \
--secret id=npm_token,src=./npm-token.txt \
-t registry.bank.local/transfer-service:abc1234 .
What are common mistakes?
# WRONG: echoing a secret in a debug statement
echo "Deploying with password: $DB_PASSWORD" # Appears in logs!
# CORRECT: never echo secrets. Masking is best-effort.
echo "Deploying transfer-service to prod"
# WRONG: same credentials for QA and Prod
# A leaked QA credential can touch production.
# CORRECT: per-environment credentials with least privilege
# QA gets qa-deploy-token, Prod gets prod-deploy-token
What should a senior engineer know?
- Least privilege: each pipeline and each environment gets its own narrowly scoped credentials.
- Audit logging: the secret store logs every access so you can answer who read the production password and when.
- Treat any secret that might have hit a log as compromised and rotate it immediately.
- Mounted files are preferred over environment variables because env vars leak more easily into logs, crash dumps, and child processes.
- Rotate secrets on a schedule and immediately if exposed.
# Senior pattern: rotation check
vault read secret/data/banking/transfer
# Check last_rotated field, alert if older than policy allows.
Visual Overview
Where secrets must NEVER be:
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Source Code │ │ Build Logs │ │ Image Layers │
│ (Git) │ │ (Console) │ │ (Registry) │
│ NEVER! │ │ NEVER! │ │ NEVER! │
└──────────────┘ └──────────────┘ └──────────────┘
Where secrets live:
┌─────────────────────────────┐
│ Secret Store │
│ (Vault / AWS SM / CI store)│
└──────────────┬──────────────┘
│ fetch at runtime
▼
┌─────────────────────────────┐
│ Pipeline Step │
│ Masked env var or mounted │
│ file, scoped to one step │
└──────────────┬──────────────┘
│
▼
┌─────────────────────────────┐
│ Running Container │
│ K8s Secret mount or │
│ External Secrets Operator │
└─────────────────────────────┘
Key Takeaway
Managing secrets in a CI/CD pipeline comes down to keeping sensitive values out of three places where they tend to leak, and fetching them from one place where they are safe. The three dangerous places are source code (Git history), build logs (console output), and image layers (Docker registry). The safe place is a dedicated secret store like Vault, AWS Secrets Manager, or the CI system's encrypted store.
Concretely, I never commit secrets to Git and I wire gitleaks into CI to fail the build if one is ever committed. I never bake secrets into images because every layer is readable and persistent. When the pipeline needs a secret, it fetches it from the store at runtime, injects it as a masked variable scoped to one step, and the running application reads it from a mounted Kubernetes Secret synced from the real store.
The strongest pattern is having no static secret at all. The pipeline authenticates with OIDC federation, exchanging a signed identity token for temporary credentials that expire in minutes. Where static secrets are unavoidable, each environment gets its own narrowly scoped credentials, rotated on schedule and immediately if exposed.
Remember one sentence: secrets never live in code, logs, or images; they are fetched at runtime from a dedicated store and injected as masked, scoped, short-lived credentials.
---