Scenario: Cannot SSH into an EC2 Instance

• Instance State – Confirm the instance is running and both system and instance status checks pass; a failed check means the box may not have booted.
• Security Group – The stateful firewall must allow inbound TCP port 22 from your source IP; stale IP rules are the most common cause of SSH failure.
• NACL – The stateless subnet firewall must allow both inbound port 22 and outbound ephemeral ports (1024-65535) for the response.
• Route Table / IGW – For public instances, the route table must have 0.0.0.0/0 pointing to an Internet Gateway, and the instance needs a public or Elastic IP.
• Key Pair / OS Firewall – The correct .pem file, correct username, locked-down permissions, and no internal firewall (iptables, firewalld) blocking port 22.

What is the systematic troubleshooting approach?

When you cannot SSH into an EC2 instance, do not guess. Walk the packet's path layer by layer, from the instance outward, because the connection has to pass through every layer and it is being dropped at exactly one of them. Check in order: instance state, Security Group, NACL, route table, public IP, key pair, OS firewall.

# Quick diagnostic checklist
aws ec2 describe-instance-status --instance-id i-0abc  # running + checks passed?
aws ec2 describe-security-groups --group-ids sg-0abc    # inbound 22 from my IP?
curl -s https://checkip.amazonaws.com                   # my current public IP
ssh -vvv -i key.pem ec2-user@<public-ip>                # verbose shows where it stalls

Is the instance running and healthy?

Start at the instance itself. Confirm it is in the running state and that both status checks pass. A failed system or instance status check means the box may not even be booted or the SSH daemon may not be up, so nothing network-side matters yet. The EC2 console's system log shows boot output.

# Check instance state and status
aws ec2 describe-instance-status --instance-id i-0abc
# Look for: InstanceState=running, SystemStatus=ok, InstanceStatus=ok

# If status checks fail, get the boot log
aws ec2 get-console-output --instance-id i-0abc --output text
# Look for kernel panic, disk full, or sshd startup failure

Is the Security Group allowing port 22?

The inbound rules must allow TCP port 22 from your source IP. A very common mistake is an office IP that changed, or a rule allowing only an old CIDR. Remember the group is stateful, so you only need the inbound rule; the response returns automatically.

# Check Security Group inbound rules
aws ec2 describe-security-groups --group-ids sg-0abc \
  --query 'SecurityGroups[*].IpPermissions[?FromPort==`22`]'

# Verify your current IP matches the allowed CIDR
curl -s https://checkip.amazonaws.com
# If your IP is 203.0.113.50 but the rule allows 198.51.100.0/24, that is the problem

Is the NACL allowing both directions?

Because NACLs are stateless, you need an inbound allow for port 22 from your IP and an outbound allow for the ephemeral port range (1024-65535) so the response can leave. Forgetting the outbound ephemeral rule is a classic cause of a connection that hangs.

# NACL rules needed for SSH to work:
Inbound  Rule 100: Allow TCP 22   from 0.0.0.0/0     (or your specific IP)
Outbound Rule 100: Allow TCP 1024-65535 to 0.0.0.0/0  (ephemeral return ports)

# If outbound ephemeral is missing:
# SSH connects, sends packets, but responses are dropped
# Connection hangs indefinitely (not refused, just silent)

Does the instance have network reachability?

For a public instance there must be a route table entry sending 0.0.0.0/0 to an Internet Gateway, and the instance must actually have a public IP or Elastic IP. If it only has a private IP, you cannot reach it directly from the internet and must go through a bastion host or SSM Session Manager.

# Check if the instance has a public IP
aws ec2 describe-instances --instance-id i-0abc \
  --query 'Reservations[*].Instances[*].[PublicIpAddress,PrivateIpAddress,SubnetId]'

# Check the subnet's route table for IGW route
aws ec2 describe-route-tables --filters "Name=association.subnet-id,Values=subnet-0abc" \
  --query 'RouteTables[*].Routes[?DestinationCidrBlock==`0.0.0.0/0`]'
# Should show GatewayId: igw-xxxx for a public subnet

Is the key pair correct?

Are you using the right .pem file for the key pair attached at launch, the right username for the AMI (ec2-user, ubuntu, admin), and are the key file permissions locked down?

# WRONG: permissions too open
$ ssh -i nbi-key.pem ec2-user@54.123.45.67
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for 'nbi-key.pem' are too open.

# CORRECT: fix permissions
chmod 400 nbi-key.pem
ssh -i nbi-key.pem ec2-user@54.123.45.67  # Amazon Linux default user
# For Ubuntu: ubuntu@...
# For Debian: admin@...

Could the OS firewall be blocking?

Even if all AWS layers are correct, an inside-the-OS firewall such as iptables, firewalld, or a misconfigured ufw can block port 22, and sshd might not be running or might listen on a non-standard port.

# If you can reach the instance through SSM, check from inside:
sudo systemctl status sshd          # Is SSH daemon running?
sudo iptables -L -n | grep 22       # Is iptables blocking port 22?
sudo firewall-cmd --list-all        # Is firewalld blocking?
sudo ss -tlnp | grep 22             # Is sshd actually listening on port 22?

How does ssh -vvv help diagnose?

The ssh -vvv command shows verbose connection details. The stall point tells you the category of failure:

• Stalls at "Connecting to..." – The packet is being dropped by a network layer (Security Group, NACL, route, or no public IP). No TCP handshake completed.
• Connects but "Connection refused" – TCP reached the instance but sshd is not listening on that port.
• Connects but "Permission denied (publickey)" – The network is fine. The problem is the key, username, or OS-side authorized keys.

ssh -vvv -i nbi-key.pem ec2-user@54.123.45.67
# Watch the output:
# debug1: Connecting to 54.123.45.67 [54.123.45.67] port 22.
# (hangs here = network problem)
# debug1: Connection established.
# (reaches here = network OK, check auth)
# debug1: Trying private key: nbi-key.pem
# Permission denied (publickey).
# (wrong key or wrong username)

What are common mistakes?

The most common mistake is a changed source IP. If you work from home and your ISP changes your public IP, the Security Group rule for the old IP blocks you silently.

The second mistake is trying to SSH to a private instance directly from the internet. A private instance has no public IP and no IGW route; SSH from the internet is impossible by design.

The third mistake is checking only the source side. SSH is a two-way conversation: the Security Group handles return traffic automatically (stateful), but the NACL does not (stateless), so you must check both directions on the NACL.

public class SSHTroubleshootingOrder {
    enum CheckOrder {
        INSTANCE_STATE,      // Is it running? Status checks pass?
        SECURITY_GROUP,      // Inbound TCP 22 from my IP?
        NACL,                // Inbound 22 + outbound ephemeral?
        ROUTE_TABLE,         // 0.0.0.0/0 to IGW?
        PUBLIC_IP,           // Has public or Elastic IP?
        KEY_PAIR,            // Right .pem, right username, right permissions?
        OS_FIREWALL          // iptables, firewalld, sshd running?
    }
}

What should a senior engineer know?

A senior engineer uses VPC Reachability Analyzer to programmatically trace the path and find the exact hop where traffic is blocked, instead of manually checking each layer. They also know that for banking production systems, SSH should be the emergency break-glass path, not the normal access method. The default should be SSM Session Manager with no inbound ports open and full CloudTrail logging.

# VPC Reachability Analyzer – finds the exact block point
aws ec2 create-network-insights-path \
  --source i-0myworkstation \
  --destination i-0abc \
  --protocol tcp \
  --destination-port 22

aws ec2 start-network-insights-analysis \
  --network-insights-path-id nip-0abc
# Returns the exact component blocking the path