Definition
- Resource monitoring – A shell script that reads CPU, memory, and disk utilization of a host and compares each against a configured threshold such as 80%.
- Alerting – When any metric crosses the threshold, the script logs a timestamped warning and optionally sends a notification (webhook, email, Slack).
- Exit code – Returns zero when everything is healthy and non-zero on any breach, so it can gate automation like deployment pipelines or readiness probes.
- Defensive scripting – Uses
set -euo pipefail, quoted variables, and tool-existence checks to prevent silent failures. - Configurability – Thresholds, filesystem paths, and alert channels are overridable through environment variables.
Think of a night-watchman at the National Bank of India who every few minutes checks three gauges: how hard the computers are working (CPU), how full their short-term memory is (RAM), and how full the filing cabinets are (disk). If any gauge goes past the red line at 80%, the watchman rings a bell and writes it in the logbook so the on-call engineer can act before the system chokes mid-transaction for Rajesh or Priya.
Understanding the Concepts
What does the script need to do?
The script reads three utilization metrics, compares each against a threshold, and alerts if any is over. It must be safe to run unattended on a cron schedule, so it needs defensive coding, configurable thresholds, and a meaningful exit code.
# High-level steps:
# 1. Set defaults (threshold, filesystem, log file)
# 2. Read CPU utilization
# 3. Read memory utilization
# 4. Read disk utilization
# 5. Compare each to threshold
# 6. Alert and log if any is breached
# 7. Exit 0 (healthy) or exit 1 (breached)
How do you read CPU utilization?
On Linux, derive busy percentage as 100 minus the idle percentage reported by top in batch mode, or parse /proc/stat directly. The top -bn1 approach is simple and portable.
# Method 1: Using top in batch mode
CPU_IDLE=$(top -bn1 | grep "Cpu(s)" | awk '{print $8}' | cut -d'.' -f1)
CPU_USED=$((100 - CPU_IDLE))
# Method 2: Using /proc/stat (more reliable in containers)
# Read two snapshots 1 second apart and calculate the delta
read_cpu() {
local line
line=$(head -1 /proc/stat)
echo "$line" | awk '{print $2+$3+$4, $5}'
}
SNAP1=$(read_cpu)
sleep 1
SNAP2=$(read_cpu)
CPU_USED=$(echo "$SNAP1 $SNAP2" | awk '{
busy1=$1; idle1=$2; busy2=$3; idle2=$4;
total=((busy2-busy1)+(idle2-idle1));
printf "%d", ((busy2-busy1)*100/total)
}')
How do you read memory utilization?
Use free and compute used over total from the "Mem:" line. Use the available column rather than raw free to measure real pressure, because Linux uses free memory for cache and buffers.
# Read memory utilization
MEM_TOTAL=$(free | awk '/^Mem:/ {print $2}')
MEM_AVAILABLE=$(free | awk '/^Mem:/ {print $7}')
MEM_USED_PCT=$(( (MEM_TOTAL - MEM_AVAILABLE) * 100 / MEM_TOTAL ))
How do you read disk utilization?
Read the use percentage of the target filesystem from df -P. The -P flag ensures POSIX output format, which is consistent across systems.
# Read disk utilization for the data volume
DISK_FS="${DISK_FS:-/}"
DISK_USED_PCT=$(df -P "$DISK_FS" | awk 'NR==2 {gsub(/%/,""); print $5}')
What does the complete script look like?
Here is a production-quality script with all the pieces together.
#!/usr/bin/env bash
# ============================================================
# check_resources.sh
# Checks CPU, memory, and disk utilization for National Bank
# of India transaction hosts. Alerts if any exceeds threshold.
# ============================================================
set -euo pipefail
# --- Configuration (overridable via environment variables) ---
THRESHOLD="${THRESHOLD:-80}"
DISK_FS="${DISK_FS:-/}"
LOG_FILE="${LOG_FILE:-/var/log/resource-check.log}"
WEBHOOK_URL="${WEBHOOK_URL:-}"
# --- Helper: check required tools exist ---
for cmd in top free df awk; do
if ! command -v "$cmd" >/dev/null 2>&1; then
echo "ERROR: Required tool '$cmd' not found." >&2
exit 3
fi
done
# --- Read CPU utilization ---
CPU_IDLE=$(top -bn1 | grep "Cpu(s)" | awk '{print $8}' | cut -d'.' -f1)
CPU_USED=$((100 - CPU_IDLE))
# --- Read memory utilization ---
MEM_TOTAL=$(free | awk '/^Mem:/ {print $2}')
MEM_AVAILABLE=$(free | awk '/^Mem:/ {print $7}')
MEM_USED_PCT=$(( (MEM_TOTAL - MEM_AVAILABLE) * 100 / MEM_TOTAL ))
# --- Read disk utilization ---
DISK_USED_PCT=$(df -P "$DISK_FS" | awk 'NR==2 {gsub(/%/,""); print $5}')
# --- Compare and alert ---
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
BREACHED=0
alert() {
local metric="$1" value="$2"
local msg="$TIMESTAMP ALERT: $metric at ${value}% (threshold: ${THRESHOLD}%)"
echo "$msg" | tee -a "$LOG_FILE"
# Optional: send to webhook
if [[ -n "$WEBHOOK_URL" ]]; then
curl -s -X POST "$WEBHOOK_URL" \
-H 'Content-Type: application/json' \
-d "{\"text\": \"$msg\"}" >/dev/null 2>&1 || true
fi
BREACHED=1
}
if (( CPU_USED > THRESHOLD )); then
alert "CPU" "$CPU_USED"
fi
if (( MEM_USED_PCT > THRESHOLD )); then
alert "MEMORY" "$MEM_USED_PCT"
fi
if (( DISK_USED_PCT > THRESHOLD )); then
alert "DISK($DISK_FS)" "$DISK_USED_PCT"
fi
# --- Report healthy metrics ---
if (( BREACHED == 0 )); then
echo "$TIMESTAMP OK: CPU=${CPU_USED}% MEM=${MEM_USED_PCT}% DISK=${DISK_USED_PCT}%" \
| tee -a "$LOG_FILE"
fi
# --- Exit code: 0=healthy, 1=breached ---
exit "$BREACHED"
How do you run it on a schedule?
Add a cron entry to run every five minutes. The exit code can also be used as a health-check gate in a CI/CD pipeline.
# Cron: run every 5 minutes
# crontab -e
*/5 * * * * /opt/scripts/check_resources.sh >> /var/log/resource-check.log 2>&1
# Or as a systemd timer for better logging
# /etc/systemd/system/resource-check.timer
# Expected output when healthy:
2024-01-15 10:30:00 OK: CPU=45% MEM=62% DISK=55%
# Expected output when a threshold is breached:
2024-01-15 10:35:00 ALERT: MEMORY at 87% (threshold: 80%)
What are common mistakes? (WRONG then CORRECT)
# WRONG: No set -euo pipefail
# A failed command silently continues, and the script reports "healthy"
# even though it could not read CPU or memory.
#!/bin/bash
CPU_USED=$(some_broken_command) # fails silently, CPU_USED is empty
if (( CPU_USED > 80 )); then ... fi # comparison fails or skips
# CORRECT: set -euo pipefail catches errors immediately
#!/usr/bin/env bash
set -euo pipefail
CPU_USED=$(some_broken_command) # script exits immediately with an error
# WRONG: Using unquoted variables
DISK_FS=/data volume # word-splitting breaks the path
df $DISK_FS # df sees two arguments: /data and volume
# CORRECT: Always quote variable expansions
DISK_FS="/data volume"
df -P "$DISK_FS" # df sees one argument: /data volume
What should a senior engineer know?
A senior engineer keeps the script idempotent and side-effect free apart from logging and alerting. They log in a consistent, machine-parseable format (e.g., JSON) so a log shipper like Fluentd or Filebeat can index the alerts. They also know that this script is a lightweight supplement, not a replacement for proper monitoring with Prometheus and Grafana, but it is invaluable as a last-resort check on hosts where agents are not installed or as a readiness gate.
# JSON-format logging for machine parsing
alert_json() {
local metric="$1" value="$2"
printf '{"timestamp":"%s","level":"ALERT","metric":"%s","value":%d,"threshold":%d}\n' \
"$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$metric" "$value" "$THRESHOLD" \
| tee -a "$LOG_FILE"
}
What are common interview follow-up questions?
- "How is this different from CloudWatch or Prometheus?" – This is a lightweight, agent-free check that works on any Linux host. CloudWatch and Prometheus are proper monitoring systems with dashboards, history, and alerting rules, but they require agents or exporters.
- "How would you make the alert channel pluggable?" – Accept a WEBHOOK_URL or EMAIL environment variable, and call the appropriate notification command based on which is set.
- "Can this script be used as a Kubernetes readiness probe?" – Yes, if it returns exit code 0 the probe passes; exit code 1 fails the probe, removing the Pod from Service endpoints.
Visual Overview
Resource Check Script Flow
===========================
┌─────────────────┐
│ set -euo │ Defensive defaults
│ pipefail │ Check tools exist
└────────┬────────┘
│
┌─────┴─────┬───────────┐
▼ ▼ ▼
┌──────┐ ┌────────┐ ┌──────┐
│ CPU │ │ MEMORY │ │ DISK │
│ top │ │ free │ │ df │
│ -bn1 │ │ (avail)│ │ -P │
└──┬───┘ └───┬────┘ └──┬───┘
│ │ │
▼ ▼ ▼
┌──────────────────────────────┐
│ Compare each to THRESHOLD │
│ (default: 80%) │
└──────────┬───────────────────┘
│
┌────┴────┐
│ Any │
│ over? │
└────┬────┘
/ \
yes no
│ │
▼ ▼
┌──────────┐ ┌──────────┐
│ ALERT │ │ OK │
│ Log + │ │ Log │
│ Webhook │ │ │
│ exit 1 │ │ exit 0 │
└──────────┘ └──────────┘
Key Takeaway
The resource-monitoring script reads CPU, memory, and disk utilization and compares each against a configurable threshold, defaulting to 80%. For CPU it derives busy percent from top in batch mode; for memory it computes used over total from free using the available column to measure real pressure; and for disk it reads the use percent from df. When any metric is breached, it writes a timestamped warning to an append-only log and optionally posts to a webhook so the on-call engineer is paged.
The script is written defensively because it runs unattended on a tight cron schedule. It uses set -euo pipefail so it fails loudly instead of silently continuing, quotes every variable expansion to avoid word-splitting, and verifies required tools exist before using them. The threshold, filesystem, and alert channel are configurable through environment variables so the same script serves different hosts. Crucially, the exit code reflects health: zero when healthy and non-zero on any breach, so the script also works as a readiness gate in a deployment pipeline. At the National Bank of India, this script runs every five minutes on each transaction host, catching disk and memory pressure before they can cause the transfer service to fail for Rajesh and Priya.
Remember one sentence: read CPU, memory, and disk, compare each to the threshold, alert and exit non-zero on any breach, and keep the script defensive, configurable, and idempotent.
---