Configuration Drift: What It Is, How to Detect It, and How to Reconcile It | DevOps, CI/CD & Tooling

Q: How do you detect drift?

terraform plan detects drift by refreshing the state against reality and showing any differences as proposed changes. Standard plan shows drift as proposed changes terraform plan aws_db_instance.ledger: ~ instance_class = "db.m5.2xlarge" "db.m5.large" This means reality is db.m5.2xlarge but config s

Q: What is a refresh-only apply?

A refreshonly apply updates the state to match reality without changing any resources. It is a narrower tool to realign state when you want to accept reality asis. Refreshonly apply: update state to match reality terraform apply refreshonly This does NOT change any cloud resources It only updates th

Q: How do you prevent drift?

Prevention is procedural, not just technical. 1. Forbid manual console changes to Terraformmanaged resources Use IAM policies that restrict direct modification Only the CI pipeline role can modify managed resources 2. Route all changes through the pipeline Every infrastructure change goes through a

Definition

Drift is reality diverging from the declared configuration due to out-of-band changes; detect it with terraform plan and reconcile by re-applying, updating the config to match, or importing untracked resources.

Configuration drift is the gap between the real infrastructure and what Terraform's configuration and state describe.

Cause – changes made outside Terraform: a manual console edit, an automated process, or another tool.
Detection – terraform plan refreshes state against reality and shows differences as proposed changes.
Reconciliation – re-apply (push reality back to code), update config (adopt the change), or terraform import (bring an untracked resource under management).
Prevention – forbid manual changes, route everything through the pipeline, run scheduled drift detection.

At the National Bank of India, the official blueprint and inventory ledger say the vault door has a specific lock. One night a technician manually swaps the lock without updating the ledger. Now the blueprint and the real vault disagree, which is drift. You find it by walking the branch with the ledger in hand and comparing, which is what terraform plan does. You fix it by restoring the old lock, updating the blueprint to accept the new lock, or recording a newly installed safe into the ledger.

Understanding the Concepts

What causes configuration drift?

Drift happens because Terraform is not the only actor that can change cloud resources. Any out-of-band change makes the real resource differ from what Terraform's state and configuration describe.

BASH

# Common causes of drift in a banking environment:

# 1. Manual console edit during an incident
# An on-call engineer bumps the database instance class from db.m5.large to db.m5.2xlarge
# Terraform still says db.m5.large

# 2. Auto-scaling process changes a setting
# AWS Auto Scaling modifies a launch configuration attribute

# 3. Another team's tool edits a security group
# A compliance script adds a firewall rule directly

# 4. Provider default changes
# AWS changes the default encryption setting for new S3 buckets

How do you detect drift?

terraform plan detects drift by refreshing the state against reality and showing any differences as proposed changes.

BASH

# Standard plan - shows drift as proposed changes
terraform plan
# aws_db_instance.ledger:
#   ~ instance_class = "db.m5.2xlarge" -> "db.m5.large"
#   # This means reality is db.m5.2xlarge but config says db.m5.large
#   # Plan proposes to change it back to db.m5.large

# Refresh-only plan - shows drift without proposing fixes
terraform plan -refresh-only
# Note: Objects have changed outside of Terraform
# aws_db_instance.ledger:
#   ~ instance_class = "db.m5.large" -> "db.m5.2xlarge"
#   # This shows what ACTUALLY changed in reality

How do you reconcile drift?

There are three strategies, and choosing correctly is the real skill.

BASH

# Strategy 1: Re-apply (code wins)
# The out-of-band change was unintended -> revert to what config declares
terraform apply
# aws_db_instance.ledger will be changed:
#   instance_class: db.m5.2xlarge -> db.m5.large
# Apply? yes
# The manual change is reverted

HCL

# Strategy 2: Update config (reality wins)
# The out-of-band change was actually desired (emergency scale-up)
# Codify it so code and state agree
resource "aws_db_instance" "ledger" {
  instance_class = "db.m5.2xlarge"   # updated to match reality
}

BASH

terraform plan
# No changes. Infrastructure is up-to-date.
# Config, state, and reality all agree now

BASH

# Strategy 3: terraform import (untracked resource)
# A security group exists in AWS but is not in the state at all
terraform import aws_security_group.compliance sg-0abc123def456
# Importing... done

# Now add the matching resource block to the config
terraform plan
# No changes after import + config alignment

How do you set up scheduled drift detection?

Run a plan in CI on a schedule. If the plan shows changes, alert the team.

YAML

# GitHub Actions: scheduled drift detection
name: Drift Detection
on:
  schedule:
    - cron: "0 8 * * *"   # every morning at 8 AM

jobs:
  detect-drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Terraform Init
        run: terraform init -backend-config=prod.hcl
        working-directory: environments/prod

      - name: Detect Drift
        run: |
          terraform plan -refresh-only -detailed-exitcode \
            -var-file=prod.tfvars 2>&1 | tee plan-output.txt
          # Exit code 2 means drift detected
        working-directory: environments/prod

      - name: Alert on Drift
        if: failure()
        run: |
          echo "Drift detected in production!"
          # Send Slack notification

What is a refresh-only apply?

A refresh-only apply updates the state to match reality without changing any resources. It is a narrower tool to realign state when you want to accept reality as-is.

BASH

# Refresh-only apply: update state to match reality
terraform apply -refresh-only
# This does NOT change any cloud resources
# It only updates the state file to reflect what actually exists

# After this, a normal plan shows the diff between config and the new state
terraform plan
# aws_db_instance.ledger:
#   ~ instance_class = "db.m5.2xlarge" -> "db.m5.large"
# Config still says db.m5.large, but state now agrees with reality

How do you prevent drift?

Prevention is procedural, not just technical.

BASH

# 1. Forbid manual console changes to Terraform-managed resources
# Use IAM policies that restrict direct modification
# Only the CI pipeline role can modify managed resources

# 2. Route all changes through the pipeline
# Every infrastructure change goes through a PR -> plan -> review -> apply cycle

# 3. Tag Terraform-managed resources
# So operators know not to touch them manually

HCL

# Tag resources as Terraform-managed
resource "aws_db_instance" "ledger" {
  tags = {
    ManagedBy   = "terraform"
    Environment = "prod"
    DoNotEdit   = "true"
  }
}

What are common mistakes?

The most common mistake is blindly applying after detecting drift. In a bank, blindly reverting an emergency fix or blindly adopting an unauthorized change can both be dangerous.

BASH

# WRONG - detect drift, blindly apply
terraform plan
# Shows: instance_class db.m5.2xlarge -> db.m5.large
terraform apply -auto-approve
# Just downsized the production database that was scaled up for a reason!

# CORRECT - detect drift, investigate, decide per resource
terraform plan
# Shows drift on the database instance class
# Investigation: the on-call engineer scaled up during an incident
# Decision: adopt the change into config (reality wins)

What should a senior engineer know?

Drift handling is as much about process and intent as it is about the plan command. For each drifted resource, make a conscious decision whether code should win or reality should win.

Run scheduled drift detection so surprises are caught early. In a bank, discovering drift during a high-stakes production change is the worst time to deal with it.

BASH

# Audit trail: who changed what outside Terraform
# AWS CloudTrail shows the manual API call
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=ledger-db \
  --max-results 5

Visual Overview

DIAGRAM

  Drift Detection and Reconciliation
  =====================================

  terraform plan -refresh-only
       │
       ├─→ Read state from S3
       ├─→ Query AWS for reality
       ├─→ Compare: state vs reality
       ▼
  Drift detected?
       │
       ├── No drift → "No changes. Infrastructure is up-to-date."
       │
       └── Drift found
             │
             ├─→ Strategy 1: Re-apply (code wins)
             │     terraform apply
             │     Reality reverts to config
             │
             ├─→ Strategy 2: Update config (reality wins)
             │     Edit .tf files to match reality
             │     terraform apply (no-op)
             │
             └─→ Strategy 3: Import (untracked resource)
                   terraform import <resource> <id>
                   Add matching config block


  Prevention
  ===========

  All changes
       │
       ▼
  ┌──────────────┐     ┌──────────────┐     ┌──────────────┐
  │  Git PR       │ →→→ │  terraform   │ →→→ │  terraform   │
  │  (code review)│     │  plan        │     │  apply       │
  └──────────────┘     │  (review)    │     │  (pipeline)  │
                       └──────────────┘     └──────────────┘
  Never: manual console edit on Terraform-managed resources

Key Takeaway

Configuration drift is when real infrastructure diverges from what Terraform's configuration and state describe, caused by changes made outside Terraform. An engineer might edit a setting in the console, an auto-scaling process might change something, or another team's tool might touch a resource.

Detect drift with terraform plan, which refreshes state against reality and surfaces any divergence. For inspection without intent to change, use refresh-only mode. For proactive detection, run a scheduled plan in CI that alerts the team the moment reality stops matching the declared configuration.

Reconciliation has three strategies: re-apply the existing config (code wins), update the config to match reality (the change was desired), or import an untracked resource. The real skill is choosing correctly per resource, because in a bank blindly reverting an emergency fix or blindly adopting an unauthorized change are both dangerous.

Remember one sentence: detect drift with terraform plan, then decide per resource whether code should win or reality should win, and prevent drift by routing all changes through the pipeline.

---