Definition
- Source VPC route table – must have a route for the destination CIDR pointing to the TGW attachment.
- TGW attachment – must exist and be in the available state for the source VPC.
- TGW route table association – the source attachment is associated with one TGW route table, which is its view of the network.
- Route propagation – the destination CIDR must be present in the associated TGW route table, either propagated or static.
- Return path – the destination VPC must have a route back to the source CIDR via the TGW, and the destination's TGW route table must have the source route.
- NACLs – stateless; must allow traffic in both directions on both subnets.
A letter from building A is not arriving at building C through the central hub. You do not guess -- you walk the whole delivery route. Does building A's signpost point letters for C to the hub? Is A connected to the hub? Does the hub's map list C? Do the guards at both ends allow the letter? And critically -- can C send a reply back? If the reply has no route home, the conversation dies even though the letter arrived.
Understanding the Concepts
What is the systematic troubleshooting approach?
The packet crosses many hops, and a break at any one stops it. I check them in the order the packet travels, then verify the return path.
// Systematic hop-by-hop checklist
package com.bank.troubleshooting;
import java.util.List;
public class TgwTroubleshootingChecklist {
public static final List<String> FORWARD_HOPS = List.of(
"1. Source VPC subnet route table -> route to dest CIDR via TGW",
"2. Source TGW attachment -> exists, state=available",
"3. Source attachment -> associated with which TGW route table?",
"4. TGW route table -> has route to dest CIDR (propagated or static)?",
"5. Dest security group -> allows inbound from source?",
"6. Dest NACL -> allows inbound AND outbound (stateless!)",
"7. Source NACL -> allows outbound AND inbound (stateless!)"
);
public static final List<String> RETURN_HOPS = List.of(
"8. Dest VPC subnet route table -> route BACK to source CIDR via TGW",
"9. Dest TGW attachment -> exists, state=available",
"10. Dest attachment -> associated TGW route table has source CIDR?"
);
public static void main(String[] args) {
System.out.println("=== FORWARD PATH ===");
FORWARD_HOPS.forEach(System.out::println);
System.out.println();
System.out.println("=== RETURN PATH (most commonly missed!) ===");
RETURN_HOPS.forEach(System.out::println);
}
}
How do you check the source VPC route table?
The source VPC subnet route table must have a route for the destination CIDR pointing to the Transit Gateway attachment. If this route is missing, the packet never even leaves toward the gateway.
# Source VPC subnet route table
Destination Target
10.0.0.0/16 local (VPC internal)
10.3.0.0/16 tgw-0hub (dest VPC via Transit Gateway)
172.16.0.0/16 tgw-0hub (on-prem via Transit Gateway)
// Checking route table programmatically
package com.bank.troubleshooting;
public class RouteTableCheck {
public static boolean hasRouteToDestination(String destCidr, String expectedTarget) {
// In reality: call ec2.describeRouteTables() and search routes
System.out.println("Checking: route to " + destCidr + " via " + expectedTarget);
return true; // placeholder
}
}
How do you check the TGW attachment and association?
The source TGW attachment must exist and be in the "available" state. The attachment is associated with exactly one TGW route table -- that table is the source's view of the network. Whatever routes are missing from it are destinations the source cannot reach.
// Checking TGW attachment and route table association
package com.bank.troubleshooting;
public class TgwAttachmentCheck {
public static void verify(String attachmentId) {
System.out.println("Attachment: " + attachmentId);
System.out.println(" State: available (OK) or pending/failing (PROBLEM)");
System.out.println(" Associated TGW route table: tgw-rtb-prod");
System.out.println(" Routes in tgw-rtb-prod:");
System.out.println(" 10.3.0.0/16 -> attach-shared (propagated)");
System.out.println(" 172.16.0.0/16 -> attach-onprem (propagated)");
System.out.println(" Missing route? That destination is UNREACHABLE.");
}
}
Why is the return path the most commonly missed check?
The destination VPC subnet route table must have a route back to the source CIDR via the Transit Gateway, and the destination's associated TGW route table must have the source route propagated into it. Without this, the reply has no way home.
# Both directions MUST have routes
Forward:
Source VPC RT: 10.3.0.0/16 -> tgw-0hub
Source TGW table: has 10.3.0.0/16 propagated
Return:
Dest VPC RT: 10.1.0.0/16 -> tgw-0hub <-- RETURN ROUTE
Dest TGW table: has 10.1.0.0/16 propagated <-- RETURN ROUTE
// The classic mistake: checking only the source side
package com.bank.troubleshooting;
public class ReturnPathMistake {
public static void main(String[] args) {
System.out.println("Classic mistake: engineer only checks source route table.");
System.out.println("Forward path is fine -- request arrives at destination.");
System.out.println("But destination has NO return route to source CIDR.");
System.out.println("Reply is DROPPED. TCP handshake fails.");
System.out.println("Looks like total failure, not half-working.");
System.out.println();
System.out.println("Fix: add return route in dest VPC RT and dest TGW table.");
}
}
Why do NACLs need special attention?
NACLs are stateless, unlike Security Groups. This means inbound rules do not automatically allow return traffic. Both the source and destination NACLs must explicitly allow traffic in both directions (inbound and outbound) for the connection to work.
// NACL stateless behavior
package com.bank.troubleshooting;
public class NaclStatelessExplainer {
public static void main(String[] args) {
System.out.println("Security Group: STATEFUL");
System.out.println(" Allow inbound -> return traffic automatically allowed");
System.out.println();
System.out.println("NACL: STATELESS");
System.out.println(" Allow inbound -> return traffic NOT automatically allowed");
System.out.println(" Must explicitly allow BOTH inbound and outbound");
System.out.println(" On BOTH source and destination subnets");
}
}
What tools help with TGW troubleshooting?
VPC Reachability Analyzer traces the path between two endpoints and tells you exactly which hop drops the packet. Transit Gateway Network Manager route analyzer checks whether a route exists between two attachments. These give evidence-driven debugging rather than guessing.
// Using diagnostic tools
package com.bank.troubleshooting;
public class DiagnosticTools {
public static void main(String[] args) {
System.out.println("VPC Reachability Analyzer:");
System.out.println(" Input: source ENI, destination ENI");
System.out.println(" Output: reachable or not, and WHERE the packet drops");
System.out.println();
System.out.println("TGW Network Manager Route Analyzer:");
System.out.println(" Input: source attachment/CIDR, dest attachment/CIDR");
System.out.println(" Output: whether a route path exists in TGW tables");
}
}
What is a complete troubleshooting example?
At National Bank of India, the settlement VPC could not reach shared-services through the Transit Gateway. The engineer kept adjusting the settlement route table. When I traced the whole path, the forward direction was fine -- the request arrived. But the shared-services subnet had no return route to the settlement CIDR via the gateway, so replies were dropped. Adding that return route fixed it immediately.
// Real troubleshooting scenario
package com.bank.troubleshooting;
public class RealScenario {
public static void main(String[] args) {
System.out.println("Problem: Settlement VPC cannot reach Shared-Services VPC");
System.out.println();
System.out.println("Step 1: Settlement RT has 10.3.0.0/16 -> tgw-0hub OK");
System.out.println("Step 2: Settlement attachment available OK");
System.out.println("Step 3: TGW table has 10.3.0.0/16 propagated OK");
System.out.println("Step 4: Shared-Services SG allows inbound OK");
System.out.println();
System.out.println("Step 5: Shared-Services RT has route to 10.1.0.0/16? MISSING!");
System.out.println(" -> Reply has no return path. TCP handshake fails.");
System.out.println();
System.out.println("Fix: Add 10.1.0.0/16 -> tgw-0hub to Shared-Services RT");
System.out.println("Result: Connection works immediately.");
}
}
What should a senior engineer know?
Always verify the return direction explicitly on every Transit Gateway issue. Use Reachability Analyzer and the TGW route analyzer for evidence-driven debugging rather than guessing at the source side. Understand that a forward-only path makes TCP appear to be completely dead, not half-working. Keep route tables clean with meaningful names and tags so troubleshooting is faster.
Visual Overview
TGW Troubleshooting: Forward and Return Path
FORWARD PATH (source -> destination)
┌────────────────┐ ┌─────────────────┐ ┌────────────────┐
│ Source VPC │ │ Transit Gateway │ │ Dest VPC │
│ Subnet RT │ │ │ │ Subnet RT │
│ │ │ │ │ │
│ 10.3.0.0/16 │───→│ Attachment OK? │───→│ SG allows? │
│ -> tgw-0hub │ │ Route table: │ │ NACL allows? │
│ │ │ has 10.3.0.0/16?│ │ │
└────────────────┘ └─────────────────┘ └────────────────┘
Check: association,
propagation, route
RETURN PATH (destination -> source) <-- MOST COMMONLY MISSED!
┌────────────────┐ ┌─────────────────┐ ┌────────────────┐
│ Dest VPC │ │ Transit Gateway │ │ Source VPC │
│ Subnet RT │ │ │ │ │
│ │ │ │ │ │
│ 10.1.0.0/16 │───→│ Dest TGW table: │───→│ NACL allows │
│ -> tgw-0hub │ │ has 10.1.0.0/16?│ │ inbound reply? │
│ MISSING? <<< │ │ MISSING? <<< │ │ │
└────────────────┘ └─────────────────┘ └────────────────┘
If return route is MISSING:
Request arrives at destination -> reply has no path home
TCP handshake FAILS -> looks like TOTAL failure
Key Takeaway
When a path through a Transit Gateway is broken, I trace the packet end to end through every hop in the exact order it travels, and I do it for both directions. The forward path checks the source VPC route table, the source TGW attachment, the TGW route table association and propagation, and the destination Security Groups and NACLs. Then the return path -- and this is what people forget.
The return path is the most commonly missed check. The destination VPC must have a route back to the source CIDR via the Transit Gateway, and the destination's associated TGW route table must have the source route propagated. Without this, the reply is dropped and the TCP handshake never completes, so it looks like a total failure rather than a half-working link. Engineers waste hours editing the source side when the real fault is a missing return route.
NACLs deserve special attention because they are stateless: allowing inbound does not automatically allow return traffic, unlike Security Groups. Both source and destination NACLs must explicitly allow traffic in both directions.
At National Bank of India, the settlement VPC could not reach shared-services through the Transit Gateway. The engineer kept adjusting the settlement route table. When I traced the whole path, the forward direction was fine but the shared-services subnet had no return route to the settlement CIDR. Adding that return route fixed it immediately. The senior habit is to verify the return direction explicitly on every TGW issue, and to use VPC Reachability Analyzer for evidence-driven debugging.
Remember one sentence: routing is a two-way conversation, so always verify both the forward and return paths through the Transit Gateway, because a missing return route is the single most common cause of a dead path.
---