Definition
- Pagination contract – AWS list APIs cap items per response and return a
NextToken(orMarker) when more exist; you must call repeatedly, passing the token, until no token is returned. - Boto3 paginator –
get_paginator('operation').paginate()encapsulates the token loop, transparently followingNextTokenand yielding each page until the end. - Silent failure – Ignoring the token returns partial results with no error, which is why it is the most common and dangerous automation bug in large accounts.
- Manual loop risks – Hand-rolling the token loop is error-prone: easy to start with an empty token, mishandle absent vs. empty tokens, or forget to accumulate results.
- Retry integration – Paginators also integrate with Boto3's retry configuration, retrying transient errors mid-traversal.
When the National Bank of India asks the central registry for all its accounts, the registry only mails one envelope at a time and stamps "more to follow, reference X" on it. If the bank reads the first envelope and assumes that is everything, it misses thousands of accounts and makes decisions on a partial list. A paginator is like a diligent mail clerk who automatically keeps requesting the next envelope using the reference until the registry says "that was the last one," so the bank always has the full list.
Understanding the Concepts
Why do AWS APIs paginate?
AWS APIs paginate for scalability: returning tens of thousands of items in one response would be slow, memory-heavy, and unreliable. The service caps each response (often at 50, 100, or 1000 items depending on the API) and includes a continuation token when more data exists. The contract is that you keep calling the same operation, passing the token from the previous response, until a response comes back without a token.
# The API response structure
{
'Reservations': [...], # up to ~100 items per page
'NextToken': 'abc123...' # present when more pages exist
}
# When NextToken is absent, you have reached the last page.
Why is ignoring pagination so dangerous?
It fails silently and only in large accounts. In a dev account with 20 instances, everything fits on one page, so a script that reads only the first response appears to work perfectly. Promoted to production with 2000 instances, the same script still returns without error but processes only the first page and quietly ignores the rest.
# WRONG: No pagination - works in dev, breaks silently in prod
import boto3
ec2 = boto3.client('ec2', region_name='ap-south-1')
response = ec2.describe_instances()
count = 0
for r in response['Reservations']:
for i in r['Instances']:
count += 1
print(f"Found {count} instances") # Says 87, but there are 2000!
# No error. No warning. Just a wrong answer.
# Real-world consequences in banking:
# - Cost report understates spend by 90%
# - Compliance scan misses 1900 untagged instances
# - Cleanup job skips resources, wasting thousands per month
# - All while reporting SUCCESS
How does a Boto3 paginator work?
A paginator handles the token loop correctly for you. You call get_paginator('operation_name'), then iterate paginator.paginate(), and it makes the first call, reads and resends the token, and stops at the end automatically.
import boto3
ec2 = boto3.client('ec2', region_name='ap-south-1')
paginator = ec2.get_paginator('describe_instances')
count = 0
for page in paginator.paginate():
for reservation in page['Reservations']:
for instance in reservation['Instances']:
count += 1
print(f" {instance['InstanceId']} - {instance['State']['Name']}")
print(f"Total: {count} instances") # Now correctly says 2000!
# The paginator transparently:
# 1. Makes the first call (no token)
# 2. Reads NextToken from the response
# 3. Passes it on the next call
# 4. Repeats until no NextToken remains
# 5. Yields each page for you to iterate
How is a paginator different from a manual token loop?
A manual loop can be correct but is error-prone. Common bugs include starting with an empty token that prevents the first call, mishandling absent vs. empty-string tokens, and forgetting to accumulate results across pages.
# Manual token loop (error-prone)
import boto3
ec2 = boto3.client('ec2', region_name='ap-south-1')
# BUG 1: while token loop never runs because token starts as None
token = None
instances = []
while token: # False on first iteration! Loop never executes.
response = ec2.describe_instances(NextToken=token)
for r in response['Reservations']:
for i in r['Instances']:
instances.append(i)
token = response.get('NextToken')
print(f"Found {len(instances)} instances") # Always 0!
# Manual token loop (correct but verbose)
import boto3
ec2 = boto3.client('ec2', region_name='ap-south-1')
instances = []
kwargs = {}
while True:
response = ec2.describe_instances(**kwargs)
for r in response['Reservations']:
for i in r['Instances']:
instances.append(i)
token = response.get('NextToken')
if token:
kwargs['NextToken'] = token
else:
break
print(f"Found {len(instances)} instances") # Correct, but verbose
# Paginator (correct and concise)
import boto3
ec2 = boto3.client('ec2', region_name='ap-south-1')
paginator = ec2.get_paginator('describe_instances')
instances = []
for page in paginator.paginate():
for r in page['Reservations']:
for i in r['Instances']:
instances.append(i)
print(f"Found {len(instances)} instances") # Correct and clean
How do you combine paginators with filters?
You pass the same parameters (like Filters) to paginate() as you would to the underlying API call. This gives you both server-side filtering and complete pagination.
paginator = ec2.get_paginator('describe_instances')
# Paginate with server-side filters
for page in paginator.paginate(
Filters=[
{'Name': 'tag:Environment', 'Values': ['Production']},
{'Name': 'instance-state-name', 'Values': ['running']}
]
):
for r in page['Reservations']:
for i in r['Instances']:
print(i['InstanceId'])
# Complete AND filtered: every matching instance, no extras.
What about PaginationConfig?
You can pass a PaginationConfig to control page size or limit total items, and use result_key_iterators or build_full_result() for convenience.
# Limit to 100 total items (useful for sampling)
for page in paginator.paginate(
PaginationConfig={'MaxItems': 100, 'PageSize': 50}
):
# Yields up to 2 pages of 50 items each
pass
# build_full_result() accumulates all pages into one response
full_result = paginator.paginate().build_full_result()
all_reservations = full_result['Reservations']
What are common mistakes? (WRONG then CORRECT)
| Mistake | Consequence |
|---|---|
| No pagination at all | Silent partial results in large accounts |
| while token: loop (token starts None) | Loop never executes, returns empty list |
| Forget to accumulate across pages | Each page overwrites the previous, only last page kept |
| Client-side filter after full scan | Slow, expensive, prone to throttling |
| Hardcoded page size assumption | API can return fewer items than expected |
# WRONG: Overwriting results each page instead of accumulating
for page in paginator.paginate():
instances = [] # Reset every page! Only last page survives.
for r in page['Reservations']:
for i in r['Instances']:
instances.append(i)
# CORRECT: Accumulate outside the loop
instances = []
for page in paginator.paginate():
for r in page['Reservations']:
for i in r['Instances']:
instances.append(i)
What should a senior engineer know?
Paginators also integrate with Boto3's retry configuration, so transient errors mid-pagination are retried, making the full traversal robust as well as complete. A senior engineer uses paginators for every list operation, not just EC2: S3 list_objects_v2, IAM list_users, RDS describe_db_instances, and so on. The worst bug in cloud automation is the one that returns a wrong answer with no error.
# Paginators work for all AWS list operations
s3 = boto3.client('s3')
for page in s3.get_paginator('list_objects_v2').paginate(Bucket='banking-statements'):
for obj in page.get('Contents', []):
print(obj['Key'])
iam = boto3.client('iam')
for page in iam.get_paginator('list_users').paginate():
for user in page['Users']:
print(user['UserName'])
Visual Overview
Pagination: Why It Matters
===========================
Without Pagination (WRONG)
─────────────────────────
API Response: [Page 1: 100 instances] + NextToken="abc"
Script reads: Page 1 only
Result: 100 / 2000 instances (5%!)
Status: SUCCESS (no error!)
With Boto3 Paginator (CORRECT)
──────────────────────────────
paginator.paginate()
│
▼
┌─────────┐ NextToken ┌─────────┐ NextToken ┌─────────┐ No Token
│ Page 1 │ ──────────> │ Page 2 │ ──────────> │ Page 20 │ ──> DONE
│ 100 inst│ │ 100 inst│ │ 100 inst│
└─────────┘ └─────────┘ └─────────┘
│ │ │
└───────────────────────┴───────────────────────┘
│
2000 instances
(complete!)
Key Takeaway
You must handle pagination because AWS list APIs cap items per response and signal more data by returning a NextToken. The contract is to keep calling the same operation, passing the token from the previous response, until a response comes back without a token. If you ignore it and read only the first response, you get a partial list: and the dangerous part is that this fails silently, with no error, so it is extremely common for automation to pass in a small dev account and then quietly return incomplete results in a large production account.
A Boto3 paginator solves this cleanly: call get_paginator with the operation name and iterate paginate(), and the paginator encapsulates the entire token loop, making the first call, resending the token, and stopping when no token remains. Writing the loop by hand is possible but error-prone: it is easy to use a while token loop that never runs because the token starts empty, or to forget to accumulate results across pages. At the National Bank of India, a nightly compliance scan that read only the first page passed in the test account but silently missed hundreds of untagged instances in production; switching to a paginator fixed it immediately.
Remember one sentence: always use a Boto3 paginator for every list operation because the worst automation bug is one that returns a wrong answer with no error at all.
---