Checking Logins and Inspecting Logs on EC2 Without CloudWatch | AWS & Cloud

Q: How do last and lastb show login history?

The last command reads the binary wtmp file and prints a history of successful logins and logouts, including the user, the source IP, and the session duration. The lastb command reads btmp and prints failed login attempts, which is how you spot a bruteforce attempt. Successful login history with sou

Q: How do you inspect application and service logs?

Application and service logs live under /var/log. Common files include /var/log/messages or /var/log/syslog for general system events, and applicationspecific files like /var/log/settlementservice/app.log. On systemdbased systems, journalctl queries the journal by service, time range, and priority.

Q: What are common log files on a Linux EC2 instance?

Quick overview of recent log activity ls lt /var/log/ head 20 Shows which log files were most recently modified

Q: How do you detect a brute-force attack from logs?

A bruteforce SSH attack shows as many failed login attempts from one or a few IP addresses in a short time window. You detect it by combining lastb output with auth log analysis. Count failed attempts per source IP sudo grep "Failed" /var/log/secure awk '{print $(NF3)}' sort uniq c sort rn head Outp

Definition

Logins live in the auth log and in last/lastb, current sessions in who, everything else under /var/log and journalctl, and the EC2 console system log works even when SSH does not.

Authentication Log – /var/log/auth.log (Debian/Ubuntu) or /var/log/secure (Amazon Linux/RHEL), recording every SSH login, sudo command, and failed attempt with timestamps.
last / lastb – Commands that read binary login records (wtmp and btmp) to show successful and failed login histories.
who / w – Commands that show currently logged-in users and their active processes.
journalctl – Systemd journal query tool that filters logs by service, time range, and priority.
EC2 Console System Log – Hypervisor-captured boot and kernel output, accessible even when SSH is unreachable.

CloudWatch is like National Bank of India's central security control room that collects every branch's footage in one place. But even without that central room, each branch still keeps its own visitor book at the door recording who entered and when, a separate book of rejected visitors, and its own CCTV tapes in a cupboard. So you walk to that branch and read its door register (auth.log), its entry list (last), its rejection list (lastb), and its tapes (the other logs). And if the branch door is jammed shut, head office can still peek at a snapshot of the lobby (EC2 console system log).

Understanding the Concepts

Where is the authentication log?

The authentication log is the first place to look for who logged in. On Ubuntu and Debian it is /var/log/auth.log; on Amazon Linux and Red Hat it is /var/log/secure. Every SSH login, every sudo command, and every failed password attempt is recorded there with a timestamp and source IP.

BASH

# Successful SSH logins with source IPs
sudo grep "Accepted" /var/log/secure
# Example output:
# Jan 15 10:23:45 ip-10-0-2-50 sshd[1234]: Accepted publickey for ec2-user from 10.0.1.10 port 54321

# Failed SSH attempts – possible brute-force attack
sudo grep "Failed" /var/log/secure
# Example output:
# Jan 15 10:25:12 ip-10-0-2-50 sshd[1235]: Failed password for invalid user admin from 203.0.113.50

# Privilege escalation (sudo) commands
sudo grep "sudo" /var/log/secure
# Example output:
# Jan 15 10:30:00 ip-10-0-2-50 sudo: ec2-user : TTY=pts/0 ; COMMAND=/bin/systemctl restart settlement

The last command reads the binary wtmp file and prints a history of successful logins and logouts, including the user, the source IP, and the session duration. The lastb command reads btmp and prints failed login attempts, which is how you spot a brute-force attempt.

BASH

# Successful login history with source IPs
last -a
# ec2-user pts/0  Wed Jan 15 10:23  still logged in  10.0.1.10
# ec2-user pts/0  Tue Jan 14 09:00 - 17:30 (08:30)   10.0.1.10

# Failed login attempts – brute force detection
sudo lastb -a
# admin    ssh:notty  Wed Jan 15 10:25 - 10:25 (00:00)  203.0.113.50
# root     ssh:notty  Wed Jan 15 10:25 - 10:25 (00:00)  203.0.113.50
# Many failures from one IP in seconds = brute force

# Currently logged-in sessions
who
# ec2-user pts/0  2024-01-15 10:23 (10.0.1.10)

# Active sessions with current activity
w
# USER     TTY      FROM             LOGIN@   IDLE   WHAT
# ec2-user pts/0    10.0.1.10        10:23    0.00s  tail -f /var/log/settlement.log

How do you inspect application and service logs?

Application and service logs live under /var/log. Common files include /var/log/messages or /var/log/syslog for general system events, and application-specific files like /var/log/settlement-service/app.log. On systemd-based systems, journalctl queries the journal by service, time range, and priority.

BASH

# One service's recent logs
journalctl -u settlement-service --since "1 hour ago"

# Errors since last boot
journalctl -p err -b

# Kernel messages (useful for OOM kills, disk errors)
journalctl -k --since "today"

# Follow live application output
tail -f /var/log/settlement-service/app.log

# Search for a specific transaction ID across all logs
grep -r "TXN-2024-001" /var/log/settlement-service/

How does the EC2 console system log help?

When you cannot SSH in at all, the EC2 console provides the instance system log (the serial console boot output) and an instance screenshot. These come from the hypervisor, not from inside the instance, so they work even when the box is unreachable over the network.

BASH

# Get boot/kernel log from outside the instance
aws ec2 get-console-output --instance-id i-0abc --output text
# Look for:
# - Kernel panic
# - Disk full (no space left on device)
# - sshd failed to start
# - Network interface not configured

# Get a screenshot of the console (useful for Windows or GUI issues)
aws ec2 get-console-screenshot --instance-id i-0abc

What are common log files on a Linux EC2 instance?

| Log File | Purpose |
|---|---|
| /var/log/auth.log or /var/log/secure | SSH logins, sudo, failed attempts |
| /var/log/messages or /var/log/syslog | General system events |
| /var/log/cloud-init.log | First-boot configuration (user data script output) |
| /var/log/cloud-init-output.log | stdout/stderr from user data script |
| /var/log/yum.log or /var/log/apt/history.log | Package installation history |
| /var/log/cron | Scheduled job execution |
| Application-specific logs | Whatever path the app is configured to write to |

BASH

# Quick overview of recent log activity
ls -lt /var/log/ | head -20
# Shows which log files were most recently modified

How do you detect a brute-force attack from logs?

A brute-force SSH attack shows as many failed login attempts from one or a few IP addresses in a short time window. You detect it by combining lastb output with auth log analysis.

BASH

# Count failed attempts per source IP
sudo grep "Failed" /var/log/secure | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head
# Output:
# 1523 203.0.113.50     ← clear brute force from this IP
#   12 198.51.100.25
#    3 10.0.1.10        ← internal, probably a typo

# Timeline of the attack
sudo grep "203.0.113.50" /var/log/secure | head -5
# Jan 15 10:25:01 ... Failed password for invalid user admin
# Jan 15 10:25:01 ... Failed password for invalid user root
# Jan 15 10:25:02 ... Failed password for invalid user oracle
# Hundreds of attempts per second = automated attack

What are the limitations of local logs?

Local logs have two critical limitations. First, they are lost if the instance is terminated, because the root EBS volume is deleted by default. Second, an attacker with root access can tamper with or delete local logs to cover their tracks.

That is exactly why in production we ship logs off the box to CloudWatch Logs, an ELK stack, or a SIEM. The off-box copy is durable and tamper-resistant.

BASH

# Ship logs to CloudWatch for durability and tamper resistance
# Install the CloudWatch agent
sudo yum install amazon-cloudwatch-agent -y

# Configure it to ship auth.log and application logs
# /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
# {
#   "logs": {
#     "logs_collected": {
#       "files": {
#         "collect_list": [
#           { "file_path": "/var/log/secure", "log_group_name": "nbi/auth" },
#           { "file_path": "/var/log/settlement-service/*.log", "log_group_name": "nbi/settlement" }
#         ]
#       }
#     }
#   }
# }

What should a senior engineer know?

A senior engineer knows that local logs are indispensable for live incident response and for forensics on a box without CloudWatch, but they are the first evidence source, not the only one. For a bank, the auth log and lastb output are often the first evidence in a security investigation.

They also know that log rotation (logrotate) can remove old logs, that binary log files (wtmp, btmp) can be corrupted by unexpected shutdowns, and that centralized logging with immutable storage (S3 with Object Lock) is the gold standard for audit and compliance.

BASH

# Check logrotate configuration
cat /etc/logrotate.d/syslog
# Ensure auth.log is retained long enough for audit requirements
# Banking: typically 90-365 days retention

# For forensics, preserve evidence before investigating
sudo cp /var/log/secure /tmp/secure-forensic-copy
sudo cp /var/log/wtmp /tmp/wtmp-forensic-copy
sha256sum /tmp/*-forensic-copy > /tmp/evidence-hashes.txt

Visual Overview

DIAGRAM

┌──────── EC2 Log Inspection Without CloudWatch ────────┐
│                                                        │
│  Who logged in?                                        │
│  ├── /var/log/auth.log (Ubuntu) or /var/log/secure     │
│  │   └── grep "Accepted" → successful SSH logins       │
│  │   └── grep "Failed"   → failed attempts             │
│  │   └── grep "sudo"     → privilege escalation        │
│  ├── last -a             → login history + IPs          │
│  ├── lastb -a            → failed login attempts        │
│  ├── who                 → currently logged in          │
│  └── w                   → what each user is doing      │
│                                                        │
│  What happened on the system?                           │
│  ├── journalctl -u service --since "1 hour ago"        │
│  ├── /var/log/messages or /var/log/syslog               │
│  ├── /var/log/cloud-init.log (boot config)              │
│  └── tail -f /var/log/app.log (live follow)             │
│                                                        │
│  Cannot SSH in at all?                                  │
│  ├── aws ec2 get-console-output (boot/kernel log)       │
│  └── aws ec2 get-console-screenshot (visual)            │
│  These come from hypervisor, work even when box is dead │
│                                                        │
│  Limitation: local logs lost on termination,            │
│  tamper-possible by root → ship to CloudWatch or SIEM   │
└────────────────────────────────────────────────────────┘

Key Takeaway

Even without CloudWatch, a Linux EC2 instance keeps a full record of activity locally, and knowing where to look is a core operations and incident-response skill. The first place I go to find who logged in is the authentication log – on Ubuntu and Debian it is /var/log/auth.log, and on Amazon Linux and Red Hat it is /var/log/secure. Every SSH login, every sudo command, and every failed password attempt is recorded there with a timestamp and a source IP, so grepping for Accepted shows me successful logins, grepping for Failed shows me attacks, and grepping for sudo shows privilege escalations.

Alongside the text log there are the binary login records. The last command reads wtmp and prints the history of successful logins and logouts with user, source IP, and session length, while lastb reads btmp and prints failed attempts, which is exactly how I spot a brute-force attack – many failures from one address in a short window. For live state, who shows who is logged in right now and w shows what each active user is doing. For application and system behaviour I read the files under /var/log, and on modern systemd boxes I use journalctl to query by service, time range, and priority.

There is one more tool that matters when the box is so broken I cannot even SSH in: the EC2 console system log and instance screenshot. Those come from the hypervisor, not from inside the instance, so they work even when the instance is unreachable over the network, and they often show me a kernel panic or a full disk that explains why SSH is dead.

At National Bank of India we had a legacy settlement server without CloudWatch, and when we suspected an unauthorized login I read /var/log/secure and ran lastb, which immediately showed a burst of failed attempts from one external IP – a clear brute-force signature – while last confirmed the only successful login was our own engineer from the internal network. The senior point is that local logs are lost if the instance is terminated and a root attacker can tamper with them, so in production we still ship logs off the box to CloudWatch or an ELK stack for a durable, tamper-resistant copy.

Remember one sentence: Check /var/log/secure for logins, last and lastb for history, journalctl for services, and the EC2 console log when SSH is dead.

---