Definition
- Build and tag – build the container image and tag it with the Git commit SHA for traceability.
- Push to ECR – push the image to Amazon Elastic Container Registry, the private container registry.
- Deploy – update the Kubernetes workload using
kubectl set imageorhelm upgrade --install. - Verify – run
kubectl rollout statusto block until new pods are healthy or fail the pipeline. - OIDC – GitHub Actions presents a short-lived signed token to AWS STS, which exchanges it for temporary credentials scoped to a least-privilege role.
- Why OIDC – long-lived keys never expire, must be stored, and give standing access if leaked; OIDC credentials expire in minutes and leave nothing to store.
Deploying National Bank of India's loan service is like delivering a new sealed cash cassette to an ATM. The pipeline assembles the cassette (builds the image), stores it in the secure depot (pushes to ECR), installs it into the ATM (deploys to Kubernetes), and confirms the ATM powers up correctly (rollout verification). The driver does not carry a permanent master key (long-lived AWS key); instead they show their badge (OIDC token) and receive a temporary card that expires in 15 minutes.
Understanding the Concepts
What are the four phases of the deploy pipeline?
The pipeline flows through build, push, deploy, and verify.
# Complete GitHub Actions EKS deploy pipeline
name: Deploy to EKS
on:
push:
branches: [main]
permissions:
id-token: write
contents: read
jobs:
deploy:
runs-on: ubuntu-latest
environment: production # Requires reviewer approval
steps:
# Phase 1: Build and tag with Git SHA
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
- name: Build image
run: docker build -t bank/loan-service:${{ github.sha }} .
# Authenticate via OIDC (no stored keys)
- name: Configure AWS via OIDC
uses: aws-actions/configure-aws-credentials@e3dd6a429d7300a6a4c196c26e071d42e0343502
with:
role-to-assume: arn:aws:iam::123456789012:role/bank-deploy-role
aws-region: ap-south-1
# Phase 2: Push to ECR
- name: Login to ECR
run: aws ecr get-login-password | docker login --username AWS --password-stdin 123456789012.dkr.ecr.ap-south-1.amazonaws.com
- name: Push image
run: |
docker tag bank/loan-service:${{ github.sha }} 123456789012.dkr.ecr.ap-south-1.amazonaws.com/bank-loan-service:${{ github.sha }}
docker push 123456789012.dkr.ecr.ap-south-1.amazonaws.com/bank-loan-service:${{ github.sha }}
# Phase 3: Deploy with Helm
- name: Update kubeconfig
run: aws eks update-kubeconfig --name bank-prod --region ap-south-1
- name: Helm deploy
run: |
helm upgrade --install loan-service ./chart \
--set image.tag=${{ github.sha }} \
-f chart/values-prod.yaml \
--timeout 5m \
--atomic
# Phase 4: Verify rollout
- name: Verify rollout
run: kubectl rollout status deployment/loan-service --timeout=300s
How does OIDC federation work?
GitHub Actions mints a short-lived JWT that asserts which repository and branch triggered the workflow. AWS STS validates the token and returns temporary credentials.
OIDC Flow:
GitHub Actions AWS STS AWS Resources
│ │ │
│ JWT token │ │
│ (repo, branch, │ │
│ environment) │ │
├─────────────────────▶│ │
│ │ Validate token │
│ │ Check trust policy │
│ │ (repo + branch match)│
│ Temp credentials │ │
│ (expire in 15 min) │ │
│◀─────────────────────┤ │
│ │ │
│ Push image / deploy │ │
├──────────────────────────────────────────▶│
│ │ │
// IAM trust policy -- only this repo + main branch can assume the role
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:sub": "repo:national-bank/loan-service:ref:refs/heads/main",
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
}
}
}]
}
Why does OIDC beat long-lived keys?
| Aspect | Long-Lived Access Key | OIDC Federation |
|--------|----------------------|------------------|
| Expiry | Never (until rotated) | Minutes |
| Storage | Must be stored (GitHub secret, vault) | Nothing stored |
| Leak risk | Catastrophic, standing access | Tiny window, useless after expiry |
| Rotation | Manual process, often forgotten | Not needed, fresh per run |
| Scope | Often over-privileged | Scoped to repo + branch |
| Fork protection | Fork can use the same key | Fork cannot assume the role |
Why verify the rollout?
Without verification, a broken image ships and the pipeline reports success. kubectl rollout status blocks until pods are healthy or fails if they are not.
# WRONG: deploy and walk away
kubectl set image deployment/loan-service app=bank/loan-service:abc123
# Pipeline exits 0 even if pods crash-loop
# CORRECT: deploy and verify
kubectl set image deployment/loan-service app=bank/loan-service:abc123
kubectl rollout status deployment/loan-service --timeout=300s
# Pipeline fails if pods are not healthy within 5 minutes
# With Helm --atomic: auto-rollback on failure
helm upgrade --install loan-service ./chart \
--set image.tag=abc123 \
--atomic --timeout 5m
# If new pods fail readiness, Helm rolls back to the previous revision automatically
How do you handle EKS authorization?
AWS authorization and Kubernetes authorization are separate. Assuming the IAM role gets you AWS access. To run kubectl, you must map that role to Kubernetes RBAC.
# Update kubeconfig to use the assumed role
aws eks update-kubeconfig --name bank-prod --region ap-south-1
# Modern approach: EKS Access Entries (replaces aws-auth ConfigMap)
aws eks create-access-entry \
--cluster-name bank-prod \
--principal-arn arn:aws:iam::123456789012:role/bank-deploy-role \
--type STANDARD
aws eks associate-access-policy \
--cluster-name bank-prod \
--principal-arn arn:aws:iam::123456789012:role/bank-deploy-role \
--policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
--access-scope type=namespace,namespaces=bank-prod
What are common mistakes with EKS deploys?
# WRONG: tagging images with :latest
- run: docker build -t bank/loan-service:latest .
# Cannot trace which commit is running. Kubernetes may not pull the new image.
# CORRECT: tag with Git SHA
- run: docker build -t bank/loan-service:${{ github.sha }} .
# Every image traceable to exact source. Kubernetes always pulls the new tag.
# WRONG: storing long-lived AWS keys
- uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_KEY }}
aws-secret-access-key: ${{ secrets.AWS_SECRET }}
# Static secret that never expires
# CORRECT: OIDC federation
- uses: aws-actions/configure-aws-credentials@<sha>
with:
role-to-assume: arn:aws:iam::123456789012:role/bank-deploy-role
# Nothing stored, nothing to leak
How do you gate production deploys?
Use GitHub Environments with required reviewers so a production deploy needs human approval.
jobs:
deploy:
environment: production # Requires reviewer approval before running
runs-on: ubuntu-latest
steps:
# This job only runs after a reviewer approves in GitHub
- name: Deploy to production
run: helm upgrade --install loan-service ./chart --atomic
What should a senior engineer know?
Pin action versions to commit SHAs so a compromised action cannot inject itself. Use GitHub Environments with required reviewers. Scope the IAM role to a single ECR repository and minimal EKS permissions. Always run rollout status with a timeout. If the rollout fails, Helm with --atomic rolls back automatically.
Visual Overview
EKS Deploy Pipeline Flow
==========================
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ 1. Build │───▶│ 2. Push │───▶│ 3. Deploy │───▶│ 4. Verify │
│ Tag: git SHA │ │ to ECR │ │ helm upgrade │ │ rollout │
│ │ │ │ │ --atomic │ │ status │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
│
┌──────┴──────┐
│ Healthy? │
├─── Yes ───▶ Pipeline passes
└─── No ───▶ Helm rollback
Pipeline fails
OIDC vs Long-Lived Keys
=========================
Long-lived key:
Created ──▶ Stored in GitHub ──▶ Used ──▶ [LEAKED] ──▶ Attacker has access FOREVER
OIDC:
GitHub mints JWT ──▶ AWS STS validates ──▶ Temp creds (15 min) ──▶ Expires ──▶ Useless
Key Takeaway
A Kubernetes deployment pipeline to EKS has four phases: build the image tagged with the Git commit SHA, push to ECR, deploy with Helm upgrade --install, and verify the rollout with kubectl rollout status. The verification phase is the one people forget: without it, a broken image ships and the pipeline reports success.
For authentication, OIDC federation replaces long-lived AWS keys. GitHub mints a short-lived signed token per run, AWS STS exchanges it for temporary credentials that expire in minutes. There is nothing long-lived to store, rotate, or leak, and the trust policy scopes access to a specific repository and branch so a fork cannot assume the production deploy role.
At National Bank of India we had an audit finding for a stored AWS key that had not been rotated in a year. Moving to OIDC removed the secret entirely. We also had a broken image ship because the old pipeline never checked pod health, so now every deploy ends with rollout status and Helm --atomic for automatic rollback.
Remember one sentence: authenticate with OIDC, deploy with Helm --atomic, verify with rollout status, and tag every image with the Git commit SHA.
---