Definition
These are two related disaster recovery scenarios for Terraform state.
- Deleted state – Terraform loses its map from config to real resources and tries to recreate everything; recover by restoring from S3 versioning, local backup, or rebuilding with
terraform import. - Midway failure – some resources created and recorded, others not; recover by fixing the cause and re-running apply, which continues from where it stopped.
- Targeted apply – limits the operation to a named resource or module, useful for surgical repairs.
- Golden rule – never apply against lost or inconsistent state; always restore or rebuild state first, plan, then apply.
At the National Bank of India, the inventory ledger is how the bank knows which vault and ATMs it already owns. If the ledger is shredded, the bank might try to buy a second vault, so you restore it from the head-office backup. If no backup exists, you physically walk the branch and re-enter each item into a fresh ledger. If a renovation is interrupted halfway, you compare what is physically there against the ledger and finish carefully, item by item.
Understanding the Concepts
Why is a deleted state file dangerous?
The state is Terraform's only memory of what it created. Without it, Terraform compares configuration against an empty state and concludes every resource must be created fresh.
# State exists: Terraform knows the VPC and cluster exist
terraform plan
# No changes. Infrastructure is up-to-date.
# State deleted: Terraform thinks nothing exists
terraform plan
# + aws_vpc.banking
# + aws_eks_cluster.banking
# + aws_db_instance.ledger
# Plan: 15 to add, 0 to change, 0 to destroy.
# This would create DUPLICATE production infrastructure!
How do you recover a deleted state from S3 versioning?
This is the primary recovery path. S3 object versioning keeps every write as a separate version, so a deleted or corrupted state can be restored by promoting the previous version.
# List state file versions
aws s3api list-object-versions \
--bucket nbi-terraform-state \
--prefix prod/banking-cluster/terraform.tfstate
# Restore the last good version
aws s3api get-object \
--bucket nbi-terraform-state \
--key prod/banking-cluster/terraform.tfstate \
--version-id "abc123-previous-version" \
terraform.tfstate.restored
aws s3 cp terraform.tfstate.restored \
s3://nbi-terraform-state/prod/banking-cluster/terraform.tfstate
# Verify: plan should show no changes
terraform plan
# No changes. Infrastructure is up-to-date.
# State restored successfully!
What if there is no backup?
If no backup exists, rebuild the state by importing each real resource. This is tedious but fully reconstructs the state.
# Import each resource one by one
terraform import aws_vpc.banking vpc-0abc123def456
terraform import aws_eks_cluster.banking banking-cluster
terraform import aws_db_instance.ledger ledger-db
terraform import aws_security_group.transfer sg-0def789abc123
# After importing all resources, plan should show no changes
terraform plan
# No changes. Infrastructure is up-to-date.
# If plan shows differences, adjust config to match reality
# Then re-run plan until it is clean
# Finding real resource IDs for import
aws ec2 describe-vpcs --filters Name=tag:Name,Values=banking-vpc \
--query 'Vpcs[0].VpcId' --output text
# vpc-0abc123def456
aws eks describe-cluster --name banking-cluster \
--query 'cluster.arn' --output text
# arn:aws:eks:ap-south-1:123456789:cluster/banking-cluster
aws rds describe-db-instances --db-instance-identifier ledger-db \
--query 'DBInstances[0].DBInstanceIdentifier'
# ledger-db
How do you recover from a midway apply failure?
Terraform applies resources in dependency order and records each success in state as it goes. When apply fails partway, resources before the failure are real and tracked, while the failing resource and everything after it are not.
# Apply fails midway on a quota limit
terraform apply
# aws_vpc.banking: Creation complete
# aws_subnet.private[0]: Creation complete
# aws_subnet.private[1]: Creation complete
# aws_db_instance.replica: Creating...
# Error: Error creating DB instance: DBInstanceQuotaExceeded
# State now has: VPC, subnets (created and tracked)
# Missing: replica, and everything after it
# Fix the root cause
aws service-quotas request-service-quota-increase \
--service-code rds \
--quota-code L-7B6409FD \
--desired-value 10
# Re-run apply - Terraform sees what already exists and continues
terraform apply
# aws_vpc.banking: Refreshing state... (already exists, no change)
# aws_subnet.private[0]: Refreshing state... (already exists)
# aws_db_instance.replica: Creating... complete!
# Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
How do you use targeted apply for surgical repairs?
A targeted apply limits the operation to a specific resource or module. It is useful for repairing one broken resource without risking the healthy ones.
# Only apply the broken replica, do not touch anything else
terraform apply -target=aws_db_instance.replica
# aws_db_instance.replica: Creating... complete!
# Apply complete! Resources: 1 added.
# Target a whole module
terraform apply -target=module.database
# WARNING: targeting is a surgical tool, not a normal workflow
# Always run a full plan after targeted apply to confirm consistency
terraform plan
# No changes. Infrastructure is up-to-date.
What if a midway failure created a resource that was not recorded?
If the failure left a real resource in the cloud that never got written to state, that is untracked drift. Use import to bring it under management so a retry does not duplicate it.
# The apply created a security group but crashed before recording it
# A retry would create a duplicate
# Import the orphaned resource
terraform import aws_security_group.transfer sg-0orphaned123
# Now the retry sees it in state and skips creation
terraform apply
# aws_security_group.transfer: Refreshing state... (already exists)
# No changes needed.
What is the golden rule for state disasters?
Never run a blind apply against lost or inconsistent state. An apply against an empty state duplicates production. An apply against an inconsistent state can destroy or create the wrong resources.
# WRONG - state is missing, apply blindly
terraform apply
# Creates 15 duplicate resources in production!
# CORRECT - restore state first, then plan, then apply
# Step 1: Restore state from S3 versioning
aws s3api get-object --bucket nbi-terraform-state \
--key prod/banking-cluster/terraform.tfstate \
--version-id "previous-version" restored.tfstate
# Step 2: Plan to confirm state matches reality
terraform plan
# No changes -> state restored correctly
# Step 3: Only now is it safe to apply changes
terraform apply
What are common mistakes?
The most common mistake is not having S3 versioning enabled on the state bucket. Without versioning, a deleted state object is gone forever.
# WRONG - state bucket without versioning
aws s3api get-bucket-versioning --bucket nbi-terraform-state
# Status: (empty) <- versioning is NOT enabled!
# If state is deleted, there is no recovery path except import
# CORRECT - always enable versioning on the state bucket
aws s3api put-bucket-versioning --bucket nbi-terraform-state \
--versioning-configuration Status=Enabled
What should a senior engineer know?
Back up state before risky operations like state manipulation, large refactors, or Terraform version upgrades.
Never edit state by hand except as a last resort using the terraform state commands (mv, rm, pull, push). Manual JSON edits corrupt the state.
# Safe state manipulation with Terraform commands
terraform state list # list all resources
terraform state show aws_vpc.banking # inspect one resource
terraform state mv aws_vpc.old aws_vpc.new # rename a resource
terraform state rm aws_vpc.banking # remove from state (does NOT destroy)
terraform state pull > backup.tfstate # backup current state
Visual Overview
State Disaster Recovery Decision Tree
=======================================
State file missing or corrupted?
│
├── Yes: S3 versioning enabled?
│ │
│ ├── Yes: Restore previous version
│ │ │
│ │ ▼ terraform plan → No changes? ✓ Recovered
│ │
│ └── No: Local backup (terraform.tfstate.backup)?
│ │
│ ├── Yes: Copy backup to S3
│ │ ▼ terraform plan → No changes? ✓ Recovered
│ │
│ └── No: Rebuild with terraform import
│ │
│ ▼ Import each resource by cloud ID
│ terraform plan → No changes? ✓ Recovered
│
└── No: Apply failed midway?
│
├── Fix root cause (quota, permission, bad input)
├── terraform apply → continues from where it stopped
├── Use -target for surgical repair if needed
└── Import orphaned resources if any
Golden Rule: restore or rebuild state → plan → apply
NEVER apply against empty/broken state
Key Takeaway
These are two related disaster scenarios, and the golden rule for both is: never run a blind apply against lost or inconsistent state. An apply against an empty state duplicates production infrastructure. An apply against inconsistent state can destroy or create the wrong resources.
If the state is deleted, the best recovery is to restore it from S3 object versioning, which keeps every write as a separate version. If there is no backup, rebuild the state with terraform import for each existing resource, which is tedious but fully reconstructs it. After restoring, run a plan to confirm it shows no changes.
If an apply fails midway, Terraform recorded what it created before the failure. Fix the root cause and re-run apply to continue from where it stopped. Use targeted apply to surgically repair a specific resource without touching healthy ones, and import any orphaned resources that were created but never recorded.
Remember one sentence: always restore or rebuild the state first, plan to verify it matches reality, and only then apply.
---