Back to Monitoring & Logging
Q3Question 3 of 13

What are essential shell commands for backend developers

Answer: Shell commands are text-based tools that run in a terminal, and backend developers use them daily to search logs, find files, inspect running processes, and troubleshoot production issues on Linux servers where no GUI is available.

Key Concepts
  • grep searches file contents by pattern and is the primary tool for finding errors, transaction IDs, and exceptions in log files.
  • find locates files by name, size, modification time, or type across directory trees.
  • ps and top show running processes, and combined with kill they let you manage stuck Java applications.
  • tail -f follows a log file in real time, which is essential for debugging live banking services.
  • awk and sed transform text output, enabling extraction of specific fields from structured logs.
  • Pipe (|) chains commands together, letting you filter, sort, and count results in a single line.
  • Understanding these commands is required for production troubleshooting when GUI tools are not available.

Definition

Shell commands are text-based tools that run in a terminal, and backend developers use them daily to search logs, find files, inspect running processes, and troubleshoot production issues on Linux servers where no GUI is available.
  • grep searches for text patterns in files and streams, returning matching lines with context.
  • find locates files and directories based on name, size, type, or modification time.
  • ps lists running processes with their PID, memory, CPU, and command details.
  • tail displays the end of a file, and tail -f follows new lines as they are written.
  • awk processes structured text by splitting lines into fields and applying rules.
  • sed edits text streams using pattern matching and substitution.
  • pipe (|) connects the output of one command to the input of the next, chaining operations together.

Think of a bank vault room full of filing cabinets. grep is a clerk who searches every folder for a specific account number and hands you the matching documents. find is a clerk who knows where every file is stored by name and date. ps is the security camera showing you who is currently working in the vault. tail -f is standing at the fax machine watching new transaction confirmations arrive one by one. The pipe is a conveyor belt connecting the clerks: one clerk finds the files, passes them to the next who filters, who passes to the next who counts.

Understanding the Concepts

How do you use grep to search log files?

grep is the most important command for production troubleshooting. It searches files for lines matching a pattern and prints them.

In banking production, you use grep to find specific transaction IDs, error messages, or account numbers in log files that can be gigabytes in size.

BASH
# Search for a specific transaction ID in the application log
grep "TXN-5001" /var/log/bank/transfer-service.log

# Search case-insensitive for any error
grep -i "error" /var/log/bank/transfer-service.log

# Show 3 lines before and after each match (context)
grep -B 3 -A 3 "InsufficientBalance" /var/log/bank/transfer-service.log

# Count how many failed transfers today
grep -c "TransferFailed" /var/log/bank/transfer-service.log

# Search recursively in all log files
grep -r "SAV2001" /var/log/bank/

# Search for a pattern (regex): any 500 HTTP status
grep -E "HTTP/1\.1\" 5[0-9]{2}" /var/log/bank/access.log
JAVA
package com.bank.ops;

import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;

public class LogFileGenerator {

    public static void main(String[] args) throws IOException {
        String logFile = "transfer-service.log";
        try (BufferedWriter writer = new BufferedWriter(new FileWriter(logFile))) {
            writer.write("2025-01-15 10:00:01 INFO  TransferService - TXN-5001 initiated SAV2001 -> SAV3002 amount=25000.00\n");
            writer.write("2025-01-15 10:00:02 INFO  AccountService - SAV2001 balance check: 75000.00\n");
            writer.write("2025-01-15 10:00:03 INFO  TransferService - TXN-5001 completed successfully\n");
            writer.write("2025-01-15 10:01:15 ERROR TransferService - TXN-5002 failed: InsufficientBalance SAV4003 balance=3000.00 required=10000.00\n");
            writer.write("2025-01-15 10:02:30 INFO  TransferService - TXN-5003 initiated SAV2001 -> SAV5005 amount=5000.00\n");
            writer.write("2025-01-15 10:02:31 INFO  TransferService - TXN-5003 completed successfully\n");
        }
        System.out.println("Log file generated: " + logFile);
        System.out.println("Try: grep 'ERROR' " + logFile);
        System.out.println("Try: grep 'SAV2001' " + logFile);
        System.out.println("Try: grep -c 'completed' " + logFile);
    }
}
TEXT
Log file generated: transfer-service.log
Try: grep 'ERROR' transfer-service.log
Try: grep 'SAV2001' transfer-service.log
Try: grep -c 'completed' transfer-service.log
BASH
# Running the grep commands on the generated log
grep 'ERROR' transfer-service.log
TEXT
2025-01-15 10:01:15 ERROR TransferService - TXN-5002 failed: InsufficientBalance SAV4003 balance=3000.00 required=10000.00

How do you use find to locate files?

find walks a directory tree and prints files matching your criteria. In production, you use it to locate log files, configuration files, heap dumps, or thread dumps.

BASH
# Find all .log files in /var/log/bank
find /var/log/bank -name "*.log"

# Find log files modified in the last 24 hours
find /var/log/bank -name "*.log" -mtime -1

# Find heap dump files larger than 500MB
find /opt/bank/app -name "*.hprof" -size +500M

# Find and delete old log files (older than 30 days)
find /var/log/bank -name "*.log" -mtime +30 -exec rm {} \;

# Find Java files containing "BigDecimal" (combine with grep)
find /opt/bank/app/src -name "*.java" -exec grep -l "BigDecimal" {} \;

# Find configuration files by name pattern
find /opt/bank/app -name "application-*.yml"
JAVA
package com.bank.ops;

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.stream.Stream;

public class FindFilesDemo {

    public static void main(String[] args) throws IOException {
        // Java equivalent of find: Files.walk
        Path startDir = Path.of(".");

        System.out.println("Java files in current directory:");
        try (Stream<Path> paths = Files.walk(startDir)) {
            paths.filter(p -> p.toString().endsWith(".java"))
                 .forEach(p -> System.out.println("  " + p));
        }

        System.out.println();
        System.out.println("Log files larger than 0 bytes:");
        try (Stream<Path> paths = Files.walk(startDir)) {
            paths.filter(p -> p.toString().endsWith(".log"))
                 .filter(p -> {
                     try { return Files.size(p) > 0; }
                     catch (IOException e) { return false; }
                 })
                 .forEach(p -> {
                     try {
                         System.out.println("  " + p + " (" + Files.size(p) + " bytes)");
                     } catch (IOException e) { /* skip */ }
                 });
        }
    }
}
TEXT
Java files in current directory:
  ./FindFilesDemo.java
  ./LogFileGenerator.java

Log files larger than 0 bytes:
  ./transfer-service.log (487 bytes)

How do you use ps, top, and kill to manage processes?

ps lists running processes. top shows a live, updating view of processes sorted by CPU or memory. kill sends signals to processes.

In banking production, you use these to find a stuck Java process, check its memory usage, and restart it if needed.

BASH
# List all Java processes
ps aux | grep java

# Show process tree (parent-child relationships)
ps -ef --forest | grep java

# Find the PID of the transfer service
ps aux | grep transfer-service | grep -v grep

# Check memory and CPU of a specific process
top -p 12345

# Send graceful shutdown signal
kill -15 12345

# Force kill if graceful shutdown fails
kill -9 12345

# JVM-specific: get thread dump for debugging
kill -3 12345
# or
jstack 12345 > thread-dump.txt

# JVM-specific: get heap dump
jmap -dump:format=b,file=heap.hprof 12345
JAVA
package com.bank.ops;

public class ProcessInfoDemo {

    public static void main(String[] args) {
        // Java equivalent: get current process info
        ProcessHandle current = ProcessHandle.current();

        System.out.println("Current Java process:");
        System.out.println("  PID: " + current.pid());
        System.out.println("  Command: " + current.info().command().orElse("unknown"));
        System.out.println("  User: " + current.info().user().orElse("unknown"));
        current.info().startInstant()
            .ifPresent(t -> System.out.println("  Started: " + t));

        System.out.println();
        System.out.println("JVM memory:");
        Runtime runtime = Runtime.getRuntime();
        long maxMB = runtime.maxMemory() / (1024 * 1024);
        long totalMB = runtime.totalMemory() / (1024 * 1024);
        long freeMB = runtime.freeMemory() / (1024 * 1024);
        System.out.println("  Max: " + maxMB + " MB");
        System.out.println("  Total: " + totalMB + " MB");
        System.out.println("  Free: " + freeMB + " MB");
        System.out.println("  Used: " + (totalMB - freeMB) + " MB");
    }
}
TEXT
Current Java process:
  PID: 24680
  Command: /usr/lib/jvm/java-21/bin/java
  User: bankapp
  Started: 2025-01-15T04:00:00Z

JVM memory:
  Max: 512 MB
  Total: 128 MB
  Free: 98 MB
  Used: 30 MB

How do you use tail and less for log inspection?

tail shows the last lines of a file. tail -f follows the file in real time, printing new lines as they are appended. This is the go-to command for watching a running banking service.

BASH
# Show last 50 lines of the log
tail -50 /var/log/bank/transfer-service.log

# Follow the log in real time
tail -f /var/log/bank/transfer-service.log

# Follow and filter: only show errors
tail -f /var/log/bank/transfer-service.log | grep ERROR

# Follow multiple files
tail -f /var/log/bank/transfer-service.log /var/log/bank/account-service.log

# less: navigate large files (search with /pattern)
less /var/log/bank/transfer-service.log
# Type /InsufficientBalance to search forward
# Type ?SAV2001 to search backward
# Type q to quit
JAVA
package com.bank.ops;

import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;

public class TailDemo {

    // Simulates a running application writing logs
    public static void main(String[] args) throws IOException, InterruptedException {
        String logFile = "live-transfer.log";
        try (BufferedWriter writer = new BufferedWriter(new FileWriter(logFile, true))) {
            String[] events = {
                "INFO  Rajesh logged in from 192.168.1.10",
                "INFO  TXN-6001 initiated SAV2001 -> SAV3002 amount=15000",
                "INFO  TXN-6001 completed successfully",
                "ERROR TXN-6002 failed: daily limit exceeded for SAV2001"
            };

            for (String event : events) {
                String line = java.time.LocalDateTime.now() + " " + event;
                writer.write(line);
                writer.newLine();
                writer.flush();
                System.out.println("Written: " + line);
                Thread.sleep(500);
            }
        }
        System.out.println("\nRun: tail -f " + logFile + " | grep ERROR");
    }
}
TEXT
Written: 2025-01-15T10:30:01 INFO  Rajesh logged in from 192.168.1.10
Written: 2025-01-15T10:30:01 INFO  TXN-6001 initiated SAV2001 -> SAV3002 amount=15000
Written: 2025-01-15T10:30:02 INFO  TXN-6001 completed successfully
Written: 2025-01-15T10:30:02 ERROR TXN-6002 failed: daily limit exceeded for SAV2001

Run: tail -f live-transfer.log | grep ERROR

How do you use pipes and awk to analyze logs?

Pipes connect the output of one command to the input of the next. Combined with awk (field extraction) and sort/uniq (aggregation), you can perform complex log analysis in one line.

BASH
# Count errors per service in the last hour
grep "ERROR" /var/log/bank/app.log | awk '{print $4}' | sort | uniq -c | sort -rn

# Top 10 accounts by transaction count
grep "TXN" /var/log/bank/transfer.log | awk -F'SAV' '{print "SAV"$2}' | awk '{print $1}' | sort | uniq -c | sort -rn | head -10

# Average response time from access log
awk '{sum += $NF; count++} END {print "Average:", sum/count, "ms"}' /var/log/bank/response-times.log

# Find the busiest minute
awk '{print substr($1,1,16)}' /var/log/bank/access.log | sort | uniq -c | sort -rn | head -5
JAVA
package com.bank.ops;

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.util.Map;
import java.util.TreeMap;

public class LogAnalysisDemo {

    public static void main(String[] args) throws IOException {
        // Java equivalent of: grep ERROR log | awk '{print $4}' | sort | uniq -c
        String logFile = "transfer-service.log";
        Map<String, Integer> errorCounts = new TreeMap<>();

        try (BufferedReader reader = new BufferedReader(new FileReader(logFile))) {
            String line;
            while ((line = reader.readLine()) != null) {
                if (line.contains("ERROR")) {
                    String[] parts = line.split("\\s+");
                    if (parts.length >= 4) {
                        String service = parts[3];
                        errorCounts.merge(service, 1, Integer::sum);
                    }
                }
            }
        }

        System.out.println("Error counts by service:");
        errorCounts.forEach((service, count) ->
            System.out.println("  " + count + " " + service));
    }
}
TEXT
Error counts by service:
  1 TransferService

What are common mistakes with shell commands?

Mistake 1: Using grep without context

Wrong: grep returns the error line but you have no idea what caused it.

BASH
# WRONG: just the error line, no context
grep "ERROR" transfer-service.log
TEXT
2025-01-15 10:01:15 ERROR TransferService - TXN-5002 failed: InsufficientBalance SAV4003

Correct: use -B (before) and -A (after) to see surrounding context.

BASH
# CORRECT: 3 lines before and after show the full picture
grep -B 3 -A 3 "ERROR" transfer-service.log
TEXT
2025-01-15 10:00:03 INFO  TransferService - TXN-5001 completed successfully
--
2025-01-15 10:01:15 ERROR TransferService - TXN-5002 failed: InsufficientBalance SAV4003 balance=3000.00 required=10000.00
--
2025-01-15 10:02:30 INFO  TransferService - TXN-5003 initiated SAV2001 -> SAV5005 amount=5000.00

Mistake 2: Running kill -9 as the first option

Wrong: force-killing a Java process can corrupt data, skip shutdown hooks, and leave resources open.

BASH
# WRONG: force kill immediately
kill -9 12345

Correct: send SIGTERM first, wait, then force-kill only if necessary.

BASH
# CORRECT: graceful shutdown first
kill -15 12345      # SIGTERM: allows shutdown hooks to run
sleep 30            # Wait for graceful shutdown
ps -p 12345         # Check if still running
kill -9 12345       # SIGKILL: only if still running

How do you use disk and network commands in production?

Backend developers need to check disk space (a full disk causes log writing to fail) and network connectivity (services cannot reach databases or brokers).

BASH
# Check disk usage
df -h                           # Filesystem usage summary
du -sh /var/log/bank/*          # Size of each log directory

# Check network
netstat -tlnp                   # Listening ports
ss -tlnp                        # Modern alternative to netstat
curl -v http://localhost:8080/actuator/health  # Check Spring Boot health

# Check if the database port is reachable
nc -zv db-server 5432           # Test PostgreSQL port

# DNS resolution
nslookup kafka-broker-1.bank.internal
JAVA
package com.bank.ops;

import java.io.File;

public class DiskSpaceDemo {

    public static void main(String[] args) {
        // Java equivalent of df -h
        File[] roots = File.listRoots();
        for (File root : roots) {
            long totalGB = root.getTotalSpace() / (1024 * 1024 * 1024);
            long freeGB = root.getFreeSpace() / (1024 * 1024 * 1024);
            long usedGB = totalGB - freeGB;
            int usedPct = totalGB > 0 ? (int)(usedGB * 100 / totalGB) : 0;

            System.out.printf("Disk %s: %d GB total, %d GB used (%d%%), %d GB free%n",
                root.getPath(), totalGB, usedGB, usedPct, freeGB);

            if (usedPct > 85) {
                System.out.println("  WARNING: Disk usage above 85%, clean old logs!");
            }
        }
    }
}
TEXT
Disk /: 500 GB total, 320 GB used (64%), 180 GB free

What should a senior engineer know?

- jstack for thread dumps. When a banking service is slow or stuck, jstack captures the state of every thread. Look for threads in BLOCKED or WAITING state to find deadlocks or contention on database connection pools.

- jmap for heap dumps. When the JVM is using too much memory, jmap -dump:format=b,file=heap.hprof captures the heap. Analyze it with Eclipse MAT or VisualVM to find memory leaks.

- strace for system call tracing. When a Java process is mysteriously slow, strace -p -e trace=network shows every system call it makes. This reveals DNS resolution delays, connection timeouts, or slow file I/O.

- Structured logging with jq. Modern banking services output JSON logs. Use jq to parse and filter: cat app.log | jq 'select(.level=="ERROR") | {time: .timestamp, msg: .message}'.

- Log rotation with logrotate. Without rotation, log files grow until the disk is full. Production servers use logrotate to compress and archive old logs. Java applications should use a rolling file appender in logback or log4j2.

What are common interview follow-up questions?

  • How do you find which process is using port 8080?
  • How do you search for a pattern across multiple compressed log files?
  • What is the difference between SIGTERM and SIGKILL?
  • How do you monitor disk space and set up alerts?
  • How do you trace a request across multiple services using a correlation ID in logs?

Visual Overview

DIAGRAM
  Pipe Chain: Production Log Analysis
  =====================================

  tail -f app.log | grep ERROR | awk '{print $4}' | sort | uniq -c

  ┌──────────┐    ┌─────────┐    ┌──────────┐    ┌──────┐    ┌────────┐
  │ tail -f  │───>│  grep   │───>│   awk    │───>│ sort │───>│ uniq -c│
  │ (follow  │    │ (filter │    │ (extract │    │      │    │(count) │
  │  log)    │    │ ERROR)  │    │ service) │    │      │    │        │
  └──────────┘    └─────────┘    └──────────┘    └──────┘    └────────┘

  Input:   Full log lines (thousands per second)
  Output:  "47 TransferService"
           "12 AccountService"
           " 3 NotificationService"


  Troubleshooting Decision Tree
  ==============================

  Service is slow
       │
       ├─ Check CPU: top -p <pid>
       │    └─ High CPU? -> jstack <pid> (find busy threads)
       │
       ├─ Check memory: jmap -heap <pid>
       │    └─ Full heap? -> jmap -dump (heap dump + MAT)
       │
       ├─ Check disk: df -h
       │    └─ Full disk? -> find old logs, clean up
       │
       ├─ Check network: ss -tlnp, nc -zv db 5432
       │    └─ Port unreachable? -> firewall or service down
       │
       └─ Check logs: tail -f app.log | grep ERROR
            └─ Exception found? -> trace with transaction ID

Key Takeaway

Shell commands are the Swiss Army knife of production troubleshooting. When the National Bank of India's transfer service is failing at 2 AM and there is no GUI available, you SSH into the server and work with text commands. grep finds the error in a 10 GB log file. tail -f shows you what is happening right now. ps tells you if the Java process is still running or if it crashed.

The power of shell commands comes from pipes. A single line can follow a live log, filter for errors, extract the service name, and count occurrences per service. This gives you the same insight that a log aggregation dashboard provides, but without needing Kibana or Grafana to be working.

For Java-specific troubleshooting, jstack captures thread states to diagnose deadlocks and contention. jmap captures heap dumps to find memory leaks. These tools are built into the JDK and available on every production server. Knowing how to use them is the difference between resolving an incident in minutes and escalating it for hours.

The most important habit is using context. grep without -B and -A gives you the error line but not the cause. kill -9 without trying kill -15 first skips shutdown hooks and corrupts in-flight transactions. Always start with the gentlest approach and escalate only when needed.

One sentence to remember: grep finds the problem, tail -f watches it happen, ps shows who is running, and pipes chain them all together.

---

Share
Use ← → arrow keys to navigate