Back to CI/CD Pipeline
Q34Question 34 of 56

Iterating over AWS resources with Boto3 and what a production script must add beyond the happy path

Key Concepts
  • The `describe_instances()` response has a nested `Reservations` -> `Instances` structure requiring a double loop to iterate all instances.
  • Pagination is the most critical production addition: use `get_paginator('describe_instances').paginate()` to transparently follow `NextToken` and never miss results.
  • Error handling must catch `botocore.exceptions.ClientError`, retry transient errors (throttling) with exponential backoff, and surface permission errors clearly.
  • Server-side filtering via the `Filters` parameter reduces data transferred and keeps matching authoritative instead of error-prone Python comprehensions.
  • Structured, timestamped logging of every action and result is essential for audit in a banking environment.
  • Credentials must come from the IAM role credential-provider chain, never hardcoded access keys, because hardcoded keys leak and do not rotate.

Definition

Iterate `Reservations` then `Instances` in a double loop, but in production add paginators, error handling with retries, server-side filters, structured logging, and IAM role credentials: never hardcoded keys.
  • Response structure – describe_instances() returns Reservations containing Instances lists, requiring a nested loop to reach each instance.
  • Pagination – Use get_paginator('describe_instances').paginate() to transparently follow NextToken and ensure complete results.
  • Error handling – Catch ClientError, retry transient errors (throttling, timeouts) with exponential backoff, and surface permission errors with actionable messages.
  • Server-side filtering – Pass Filters=[{'Name': 'tag:Environment', 'Values': ['Production']}] so AWS returns only matching resources.
  • Structured logging – Log timestamped progress and results for audit using Python's logging module.
  • IAM credentials – Let Boto3 use the default credential-provider chain (instance profile, IRSA) so credentials rotate automatically.

Imagine the National Bank of India asks for a list of all its data center machines. The simple clerk calls the registry once and reads the first page, but the registry only hands over one page at a time and says "there are more" (pagination), so the simple clerk misses them. A professional clerk keeps asking until there are no more pages, writes down what went wrong if the phone drops (error handling and retries), asks the registry to pre-filter to only the production machines (server-side filtering), keeps a logbook of everything done (logging), and uses the bank's official ID badge rather than a photocopied key taped to the wall (IAM role instead of hardcoded keys).

Understanding the Concepts

What is the basic response structure of describe_instances?

describe_instances() returns a dictionary with a Reservations key. Each reservation has an Instances key, and each instance is a dictionary with fields like InstanceId, State, and Tags. So the happy path is a nested loop.

PYTHON
import boto3

ec2 = boto3.client('ec2', region_name='ap-south-1')
response = ec2.describe_instances()

# Happy path: nested loop
for reservation in response['Reservations']:
    for instance in reservation['Instances']:
        instance_id = instance['InstanceId']
        state = instance['State']['Name']
        print(f"Instance: {instance_id}, State: {state}")
TEXT
# Output:
# Instance: i-0abc123, State: running
# Instance: i-0def456, State: stopped

This works for a tiny account but silently fails as the account grows.

Why is pagination the most important production addition?

The EC2 API returns at most one page of results and a NextToken when more exist. If you ignore the token, you silently process only the first page. This is the classic cause of automation that "works in dev but misses instances in prod."

PYTHON
# WRONG: No pagination - only reads the first page
response = ec2.describe_instances()
for r in response['Reservations']:
    for i in r['Instances']:
        process(i)
# If the account has 500 instances, this may only see 50-100!
PYTHON
# CORRECT: Use a Boto3 paginator
paginator = ec2.get_paginator('describe_instances')
for page in paginator.paginate():
    for reservation in page['Reservations']:
        for instance in reservation['Instances']:
            process(instance)
# Transparently follows NextToken until every page is consumed.

How do you add error handling?

AWS calls can fail transiently with throttling or timeouts, or permanently with access-denied errors. A production script catches ClientError, inspects the error code, retries transient errors, and surfaces permission errors clearly.

PYTHON
import boto3
import botocore.exceptions
import time
import logging

logger = logging.getLogger('banking-inventory')

def list_instances_with_retry(ec2_client, filters, max_retries=3):
    """List EC2 instances with pagination and retry."""
    paginator = ec2_client.get_paginator('describe_instances')
    instances = []
    for attempt in range(max_retries):
        try:
            for page in paginator.paginate(Filters=filters):
                for reservation in page['Reservations']:
                    for instance in reservation['Instances']:
                        instances.append(instance)
            return instances
        except botocore.exceptions.ClientError as e:
            error_code = e.response['Error']['Code']
            if error_code in ('Throttling', 'RequestLimitExceeded'):
                wait = 2 ** attempt
                logger.warning(f"Throttled, retrying in {wait}s (attempt {attempt + 1})")
                time.sleep(wait)
            elif error_code == 'UnauthorizedOperation':
                logger.error(f"Permission denied: {e}")
                raise
            else:
                logger.error(f"Unexpected error: {e}")
                raise
    raise RuntimeError(f"Failed after {max_retries} retries")

How do you filter server-side?

Pass a Filters argument so AWS returns only matching resources instead of downloading everything and filtering in Python. Multiple values within one filter act as OR; multiple filter entries act as AND.

PYTHON
# Server-side filter: running production instances only
filters = [
    {'Name': 'tag:Environment', 'Values': ['Production']},
    {'Name': 'instance-state-name', 'Values': ['running']}
]

paginator = ec2.get_paginator('describe_instances')
for page in paginator.paginate(Filters=filters):
    for reservation in page['Reservations']:
        for instance in reservation['Instances']:
            print(instance['InstanceId'])
# AWS returns only Production + running instances.
# No need to download thousands and filter in Python.

How do you add structured logging?

Log timestamped progress and results using Python's logging module at appropriate levels. This is essential for audit and debugging in a bank.

PYTHON
import logging

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s %(levelname)s %(name)s %(message)s',
    datefmt='%Y-%m-%dT%H:%M:%SZ'
)
logger = logging.getLogger('banking-inventory')

# Log progress
logger.info("Starting EC2 inventory for National Bank of India")
instances = list_instances_with_retry(ec2, filters)
logger.info(f"Found {len(instances)} running production instances")

for instance in instances:
    tags = {t['Key']: t['Value'] for t in instance.get('Tags', [])}
    logger.info(
        f"Instance {instance['InstanceId']}: "
        f"type={instance['InstanceType']}, "
        f"owner={tags.get('Owner', 'UNTAGGED')}"
    )
TEXT
# Log output:
# 2024-01-15T10:30:00Z INFO banking-inventory Starting EC2 inventory
# 2024-01-15T10:30:02Z INFO banking-inventory Found 47 running production instances
# 2024-01-15T10:30:02Z INFO banking-inventory Instance i-0abc123: type=m5.large, owner=rajesh

Why must credentials come from the IAM role chain?

On EC2 or EKS, the script should run under an IAM role (instance profile or IRSA) and let Boto3 pick up temporary, automatically rotated credentials. You never embed access keys in the code, because hardcoded keys leak, do not rotate, and are a serious security finding in banking.

PYTHON
# WRONG: Hardcoded access keys
import boto3
ec2 = boto3.client(
    'ec2',
    aws_access_key_id='AKIAIOSFODNN7EXAMPLE',      # leaked in Git!
    aws_secret_access_key='wJalrXUtnFEMI/K7MDENG',  # does not rotate!
    region_name='ap-south-1'
)
PYTHON
# CORRECT: Use the default credential-provider chain
import boto3
ec2 = boto3.client('ec2', region_name='ap-south-1')
# Boto3 automatically checks:
# 1. Environment variables (AWS_ACCESS_KEY_ID)
# 2. Shared credentials file (~/.aws/credentials)
# 3. IAM role via instance metadata (EC2) or IRSA (EKS)
# On EC2/EKS, it picks up temporary, auto-rotated credentials.

What does the complete production script look like?

PYTHON
#!/usr/bin/env python3
"""EC2 inventory script for National Bank of India.
Production-grade: pagination, retry, server-side filter, logging, IAM role.
"""
import boto3
import botocore.exceptions
import logging
import time

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s %(levelname)s %(name)s %(message)s'
)
logger = logging.getLogger('banking-ec2-inventory')

def list_production_instances(region='ap-south-1', max_retries=3):
    ec2 = boto3.client('ec2', region_name=region)
    paginator = ec2.get_paginator('describe_instances')
    filters = [
        {'Name': 'tag:Environment', 'Values': ['Production']},
        {'Name': 'instance-state-name', 'Values': ['running']}
    ]
    instances = []
    for attempt in range(max_retries):
        try:
            for page in paginator.paginate(Filters=filters):
                for reservation in page['Reservations']:
                    for instance in reservation['Instances']:
                        instances.append(instance)
            logger.info(f"Found {len(instances)} production instances")
            return instances
        except botocore.exceptions.ClientError as e:
            code = e.response['Error']['Code']
            if code in ('Throttling', 'RequestLimitExceeded'):
                wait = 2 ** attempt
                logger.warning(f"Throttled, retry in {wait}s")
                time.sleep(wait)
                instances.clear()
            else:
                logger.error(f"Error: {code} - {e}")
                raise
    raise RuntimeError("Max retries exceeded")

if __name__ == '__main__':
    results = list_production_instances()
    for inst in results:
        tags = {t['Key']: t['Value'] for t in inst.get('Tags', [])}
        logger.info(
            f"{inst['InstanceId']} | {inst['InstanceType']} | "
            f"owner={tags.get('Owner', 'UNTAGGED')}"
        )

What are common mistakes? (WRONG then CORRECT)

PYTHON
# WRONG: No pagination, no error handling, hardcoded keys
import boto3
ec2 = boto3.client('ec2',
    aws_access_key_id='AKIA...',
    aws_secret_access_key='secret...')
response = ec2.describe_instances()
for r in response['Reservations']:
    for i in r['Instances']:
        print(i['InstanceId'])
# Problems: misses pages, crashes on throttle, leaks keys.
PYTHON
# CORRECT: Paginator + retry + server-side filter + IAM role
ec2 = boto3.client('ec2', region_name='ap-south-1')  # IAM role
paginator = ec2.get_paginator('describe_instances')
for page in paginator.paginate(Filters=[...]):
    for r in page['Reservations']:
        for i in r['Instances']:
            logger.info(i['InstanceId'])
# Complete, resilient, secure.

What should a senior engineer know?

A senior engineer also sets an explicit region, uses least-privilege IAM policies scoped to the exact actions and resources, makes the script idempotent and read-only unless remediation is explicitly intended, and adds a --dry-run mode for any write operations. They know the difference between a demo script and a production script is pagination, error handling, server-side filtering, structured logging, and role-based credentials: the happy-path nested loop is only the beginning.

Visual Overview

DIAGRAM
Production Boto3 Script Architecture
=====================================

  ┌──────────────────────────┐
  │  IAM Role Credentials    │  No hardcoded keys
  │  (Instance Profile /     │  Temporary, auto-rotated
  │   IRSA)                  │
  └───────────┬──────────────┘
              │
              ▼
  ┌──────────────────────────┐
  │  Boto3 Paginator         │  get_paginator('describe_instances')
  │  .paginate(Filters=...)  │  Server-side filter
  └───────────┬──────────────┘
              │  (follows NextToken automatically)
              ▼
  ┌──────────────────────────┐
  │  Page 1 → Page 2 → ...  │  Every page consumed
  │  Reservations.Instances  │  No results missed
  └───────────┬──────────────┘
              │
              ▼
  ┌──────────────────────────┐
  │  Error Handling          │  Throttle → retry + backoff
  │  (ClientError catch)     │  Permission → log + raise
  └───────────┬──────────────┘
              │
              ▼
  ┌──────────────────────────┐
  │  Structured Logging      │  Timestamp, count, instance details
  │  (audit trail)           │  Machine-parseable for banking audit
  └──────────────────────────┘

Key Takeaway

To iterate EC2 instances with Boto3, call describe_instances() which returns a Reservations list containing Instances lists, so the happy path is a nested loop. That works for a tiny account, but a production script must add pagination using a Boto3 paginator so it transparently follows NextToken until every page is read, because ignoring the token is the classic reason automation works in dev but misses instances in prod.

Beyond pagination, add error handling that catches ClientError, retries transient throttling with exponential backoff, and surfaces permission errors clearly. Filter server-side with the Filters parameter so AWS returns only the resources you need, add structured timestamped logging for audit, and always use the IAM credential-provider chain so Boto3 picks up temporary, automatically rotated credentials from the instance profile or IRSA. At the National Bank of India, the EC2 inventory ran under a least-privilege IAM role, used a paginator so no instance was ever missed, retried throttles with backoff, and filtered to Production running instances server-side.

Remember one sentence: the difference between a demo and a production Boto3 script is pagination, error handling with retries, server-side filtering, structured logging, and role-based credentials.

---

Share
Use ← → arrow keys to navigate