Definition
- Toil is operational work that is manual, repetitive, automatable, tactical, devoid of enduring value, and scales linearly with service size
- It is different from overhead like meetings and email; toil is operational work tied to running the service
- Classic examples: manually restarting stuck services, hand-applying configs, responding to the same alert with the same fix
- The guideline is to keep toil below 50% of an engineer's time so the rest goes to lasting improvements
- The primary weapons are automation, self-service tooling, better defaults, and fixing root causes
At the National Bank of India, imagine a clerk who every morning manually unlocks each of five hundred ATMs one by one, types the same code into each, and writes the result in a ledger by hand. It takes three hours, it has to be redone every day, it grows worse as the bank opens more ATMs, and at the end nothing permanent has been built. That is toil. Reducing it means building a system that unlocks all ATMs with one button, freeing the clerk to design a faster transfer flow for Rajesh and Priya.
Understanding the Concepts
What defines toil?
Work is toil to the degree it matches six characteristics. Each one matters.
# The six characteristics of toil:
# 1. Manual -> A human performs the steps by hand
# 2. Repetitive -> Done over and over, not once
# 3. Automatable -> A machine could do it just as well
# 4. Tactical -> Interrupt-driven, not strategic
# 5. No enduring value -> Service is in the same state after
# 6. Scales with service -> More servers = more toil
# Classic toil examples at NBI:
# 1. Manually restarting stuck transfer-service pods every night
kubectl rollout restart deployment/transfer-service
# 2. Hand-applying the same config change across 20 services
for svc in transfer account fraud notification; do
kubectl edit configmap ${svc}-config # manual edit, 20 times
done
# 3. Manually running a release checklist
# [ ] Update version in pom.xml
# [ ] Build the artifact
# [ ] Upload to S3
# [ ] Deploy to staging
# [ ] Run smoke tests
# [ ] Deploy to production
# All manual, every single release
How is toil different from overhead?
Overhead is administrative work like meetings, email, and expense reports. Toil is specifically operational work tied to running the service.
| Aspect | Toil | Overhead |
|---|---|---|
| Type | Operational | Administrative |
| Example | Restarting stuck pods | Attending sprint planning |
| Tied to | Service size | Team size |
| Automatable | Yes | Sometimes |
| Effect of growth | More services = more toil | More people = more meetings |
Why is toil dangerous?
When toil exceeds 50%, the team is on a treadmill: all time goes to keeping the lights on and none to improvement. The toil only grows.
# Toil death spiral:
# Service grows -> More toil (linear scaling)
# Team spends 100% on toil -> No time for automation
# No automation -> Toil stays or grows
# Team burns out -> Attrition -> Remaining team has even more toil
How do you measure toil?
You cannot manage what you do not measure. Track how much time the team spends on toil by categorising on-call and operational tickets.
# Toil tracking spreadsheet:
# Task | Freq | Time/occurrence | Monthly total
# Restart stuck transfer pods | Daily | 45 min | 1350 min
# Manually provision test DBs | 10/week | 30 min | 1200 min
# Apply same config to 20 svcs | 2/month | 2 hours | 240 min
# Respond to noisy disk alert | 5/week | 15 min | 300 min
# Total monthly toil: 3090 min (51.5 hrs)
How do you reduce toil?
Attack the biggest sources in order: automation, self-service, platforms, and root-cause fixes.
# Level 1: Automate the manual task
# Before: engineer manually restarts pods nightly
# After: CronJob restarts pods automatically
apiVersion: batch/v1
kind: CronJob
metadata:
name: restart-stuck-transfers
spec:
schedule: "0 2 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: restart
image: bitnami/kubectl
command: ["kubectl", "rollout", "restart",
"deployment/transfer-service"]
# Level 2: Self-service tooling
# Before: engineer provisions test DB via ticket (30 min)
# After: developer clicks a button in internal portal (2 min)
# Level 3: Fix the root cause
# Before: pods get stuck nightly -> CronJob restarts them
# After: fix the deadlock causing pods to stick -> no restarts needed
# Eliminating the toil entirely, not just automating the symptom
What is the difference between automating symptoms and fixing root causes?
Automating a bad process just makes the bad process faster. Before automating, ask whether the task should exist at all.
# WRONG: Automate the symptom
# Alert fires nightly: "transfer-service disk full"
# -> Write a script to clean up temp files every night
# The cleanup runs forever, masking the real problem
# CORRECT: Fix the root cause
# Why is the disk filling? Temp files not being cleaned by the app
# -> Fix the app to clean temp files after processing
# -> Alert never fires again; no script needed
What are common mistakes in toil reduction?
The most common mistake is treating toil reduction as spare-time work. On a team drowning in toil, there is never spare time.
# WRONG: "We'll automate when we have time"
# There is never time on a team at 80% toil
# CORRECT: Prioritize toil reduction like any engineering work
# - Track toil in the backlog with an owner and due date
# - Allocate 20% of sprint capacity to toil reduction
# - Start with the task that consumes the most hours
What about hidden toil?
Watch for toil that is hidden inside heroics: one person quietly absorbs huge manual effort and masks the problem until they burn out or leave.
# Hidden toil at NBI:
# Pankaj (senior SRE) manually reprocesses stuck transfers
# every night for 45 minutes before anyone else is online.
# Nobody else knows about it.
# When Pankaj goes on holiday, transfers fail overnight.
# -> Make all toil visible by tracking it in tickets
What should a senior engineer know?
A senior engineer knows that some toil is healthy because it keeps engineers familiar with the system. The guideline is below 50%. They prioritize toil reduction with a backlog and owner, and they watch for hidden heroics.
What are common interview follow-up questions?
- Is all manual work toil? No. Work requiring genuine human judgement each time is not toil.
- What is the 50% guideline? Keep toil below 50% of time so the other 50% goes to engineering improvements.
- What if the team is at 90% toil? Escalate. Hire or borrow capacity to break the spiral.
Visual Overview
Toil Reduction Ladder ┌────────────────────────────────────────────┐ │ Level 3: ELIMINATE ROOT CAUSE │ │ Fix the bug so the task never exists │ ├────────────────────────────────────────────┤ │ Level 2: SELF-SERVICE │ │ Let users do it themselves safely │ ├────────────────────────────────────────────┤ │ Level 1: AUTOMATE │ │ Script the manual steps │ ├────────────────────────────────────────────┤ │ Level 0: MANUAL TOIL │ │ Human does it by hand every time │ └────────────────────────────────────────────┘ Toil Budget ┌────────────────────────────────┐ │ Engineer's time: 100% │ │ ┌──────────────┬──────────────┐│ │ │ Toil < 50% │ Engineering ││ │ │ (keep low) │ (lasting) ││ │ └──────────────┴──────────────┘│ └────────────────────────────────┘
Key Takeaway
Toil is operational work that is manual, repetitive, automatable, tactical, devoid of enduring value, and scales linearly with the size of the service. The defining test is that after you finish a piece of toil, the service is in exactly the same state as before, nothing lasting was built, and tomorrow the same work is needed again. Classic examples are manually restarting stuck services, hand-applying the same config, and responding to the same alert with the same fix.
A little toil is healthy because it keeps engineers familiar with the system, but the guideline is below 50%. Above that, the team is on a treadmill where all time goes to keeping the lights on. The first step is measuring it by categorising operational tickets to see where the hours go. Then attack the biggest sources: automation first, self-service tooling second, and fixing root causes above all.
In the National Bank of India project, an on-call engineer spent forty-five minutes every night reprocessing stuck transfers for customers like Rajesh and Priya: 1,350 minutes a month, scoring every toil characteristic. The team first automated the reprocessing with a scheduled job, then went further and added idempotent retries so transfers stopped getting stuck at all, eliminating the toil entirely rather than merely hiding it.
Remember one sentence: measure the toil, automate relentlessly, and whenever possible remove the root cause so the toil never has to exist.
---