Definition
- Pull model means Prometheus makes HTTP GET requests to each target's
/metricsendpoint on a fixed interval, controlling the load and detecting down targets immediately - Time-series database stores each unique combination of metric name and label values as a separate time-series of timestamp-value pairs
- PromQL is a functional query language for rates, aggregations, and percentiles from histograms
- Grafana connects to Prometheus as a data source and renders PromQL queries as live dashboard panels
- Alerting rules evaluate PromQL expressions on a schedule, and the
forduration ensures conditions hold continuously before firing, preventing false positives
Prometheus is like a supervisor at the National Bank of India who walks past every counter every fifteen seconds and writes down each counter's numbers: how many transfers done, how many failed, how long each took. The supervisor does the asking (pull); the counters do not shout at the supervisor (push). All those readings go into a notebook organised by counter and time (the time-series database). Grafana is the big wall display that turns the notebook into live charts. An alert is a standing instruction: "if the failure rate stays above one percent for five whole minutes, phone the on-call manager": the five-minute wait stops a single blip from waking anyone.
Understanding the Concepts
How does the Prometheus pull model work?
Prometheus uses a pull model where each application exposes an HTTP endpoint, conventionally /metrics or /actuator/prometheus in Spring Boot, that returns the current metric values in a simple text format. Prometheus is configured with a scrape interval and a list of targets.
# prometheus.yml -- scrape configuration for NBI services
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'transfer-service'
metrics_path: /actuator/prometheus
static_configs:
- targets: ['transfer-svc:8080']
- job_name: 'account-service'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
regex: account-service
action: keep
The pull model has real advantages: Prometheus controls the load, it can tell immediately if a target is down (the scrape fails), and targets do not need to know where Prometheus is.
How does the time-series database work?
Every unique combination of a metric name and its label values is a separate time-series. Each series is an append-only list of timestamp-value pairs.
# Two different time-series (different status label)
transfer_requests_total{service="transfer", status="200"}
-> [(t1, 100), (t2, 105), (t3, 112), ...]
transfer_requests_total{service="transfer", status="500"}
-> [(t1, 2), (t2, 3), (t3, 3), ...]
# Total series = metric_name x label_value_1 x label_value_2 x ...
# 3 services x 5 status codes = 15 series (fine)
# 3 services x 5 status codes x 1M accounts = 15M series (explosion!)
This is why cardinality matters: every new unique label value creates a new series, so unbounded labels like a transfer ID or account ID would create millions of series and crash the database.
What is PromQL and how do you use it?
PromQL is the functional query language built for time-series data. The most important functions are rate() for counters, histogram_quantile() for percentiles, and sum by() for aggregation.
# Requests per second over the last 5 minutes
rate(transfer_requests_total[5m])
# Error ratio for the transfer service
sum(rate(transfer_requests_total{status=~"5.."}[5m]))
/
sum(rate(transfer_requests_total[5m]))
# p99 latency from a histogram
histogram_quantile(0.99,
sum by (le) (rate(transfer_latency_seconds_bucket[5m]))
)
# Top 5 services by error rate
topk(5,
sum by (service) (rate(transfer_requests_total{status=~"5.."}[5m]))
)
# Metric types:
# Counter: only goes up (transfer_requests_total) -- use rate()
# Gauge: goes up and down (jvm_memory_used_bytes) -- use directly
# Histogram: buckets for percentiles (transfer_latency_seconds) -- use histogram_quantile()
# Summary: pre-calculated percentiles (less common)
How does Prometheus integrate with Grafana?
Grafana connects to Prometheus as a data source and lets you build panels, each backed by a PromQL query. A typical transfer service dashboard has three panels: request rate, error rate, and p99 latency.
# Grafana dashboard layout for NBI Transfer Service ┌──────────────────┬──────────────────┬──────────────────┐ │ Request Rate │ Error Rate │ p99 Latency │ │ rate(total[5m]) │ errors/total │ histogram_q() │ │ ████████████ │ ▄ │ ████████████ │ │ ████████████ │ █ │ ████████████▄▄ │ │ 300 req/s │ 0.5% │ 120ms │ └──────────────────┴──────────────────┴──────────────────┘
Grafana also supports exemplars, which link a point on a latency graph directly to a sample trace in Jaeger, bridging metrics and traces.
How do you set actionable alerts that avoid false positives?
An alerting rule is a PromQL expression plus a for duration. The for clause is the single most important tool against false positives: it means the condition must hold continuously for a specified time before the alert fires.
# alerting rules (prometheus-rules.yml)
groups:
- name: nbi-transfer-alerts
rules:
- alert: HighTransferErrorRate
expr: |
sum(rate(transfer_requests_total{status=~"5.."}[5m]))
/
sum(rate(transfer_requests_total[5m]))
> 0.01
for: 5m # Must hold for 5 minutes before firing
labels:
severity: critical
team: payments
annotations:
summary: "Transfer error rate above 1% for 5 minutes"
runbook: "https://wiki.nbi.com/runbooks/high-transfer-errors"
- alert: HighTransferLatency
expr: |
histogram_quantile(0.99,
sum by (le) (rate(transfer_latency_seconds_bucket[5m]))
) > 0.5
for: 5m
labels:
severity: warning
annotations:
summary: "Transfer p99 latency above 500ms for 5 minutes"
How is alerting on symptoms different from alerting on causes?
You alert on symptoms users feel, not on underlying causes, because a cause may be harmless while a symptom is always meaningful.
# WRONG: Alert on a cause -- high CPU might be perfectly fine
- alert: HighCPU
expr: node_cpu_seconds_total > 0.8
for: 5m
# This pages the on-call engineer, but CPU at 80% may be normal
# CORRECT: Alert on a symptom users feel
- alert: HighTransferErrorRate
expr: transfer_error_ratio > 0.01
for: 5m
# This means real users like Rajesh and Priya are seeing failures
How does Alertmanager work?
Alertmanager receives alerts from Prometheus, deduplicates, groups related alerts, silences during maintenance, and routes to the right channel.
# alertmanager.yml
route:
receiver: 'default'
group_by: ['alertname', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- match:
severity: critical
receiver: 'pagerduty-payments'
- match:
severity: warning
receiver: 'slack-payments'
receivers:
- name: 'pagerduty-payments'
pagerduty_configs:
- service_key: 'nbi-payments-key'
- name: 'slack-payments'
slack_configs:
- channel: '#nbi-payments-alerts'
What should a senior engineer know?
Every alert should be actionable and link to a runbook. If an alert fires and the on-call engineer cannot do anything about it, it is noise and should be deleted or downgraded. Multi-window multi-burn-rate alerts for error budgets page immediately on a fast burn and gently on a slow burn.
# Multi-burn-rate alert: fast burn pages immediately
- alert: TransferErrorBudgetFastBurn
expr: |
( # 1-hour burn rate
sum(rate(transfer_requests_total{status=~"5.."}[1h]))
/ sum(rate(transfer_requests_total[1h]))
) > (14.4 * 0.001) # 14.4x the budget burn rate
and
( # 5-minute burn rate
sum(rate(transfer_requests_total{status=~"5.."}[5m]))
/ sum(rate(transfer_requests_total[5m]))
) > (14.4 * 0.001)
for: 2m
labels:
severity: critical
What are common interview follow-up questions?
- Why pull instead of push? Pull lets Prometheus control load and detect down targets. Push-based systems like Graphite require targets to know where to send data.
- What is the difference between a counter and a gauge? A counter only goes up (use
rate()); a gauge goes up and down (use directly). - Why not average latency? Averages hide the tail. Use
histogram_quantile()for percentiles. - How do you handle high availability for Prometheus? Run two identical Prometheus instances scraping the same targets, and use Thanos or Cortex for long-term storage and deduplication.
Visual Overview
Prometheus Pull Model
┌───────────────┐ HTTP GET /metrics ┌───────────────┐
│ Prometheus │◀────── every 15s ────│ Transfer Svc │
│ Server │◀────── every 15s ────│ Account Svc │
│ │◀────── every 15s ────│ Fraud Svc │
└───────┬───────┘ └───────────────┘
│
▼
┌───────────────┐ PromQL ┌───────────────┐
│ Time-Series │──────────────▶│ Grafana │
│ Database │ │ Dashboards │
└───────┬───────┘ └───────────────┘
│
▼
┌───────────────┐ Routes ┌───────────────┐
│ Alert Rules │──────────────▶│ Alertmanager │
│ (PromQL) │ │ → PagerDuty │
│ for: 5m │ │ → Slack │
└───────────────┘ └───────────────┘
Key Takeaway
Prometheus works on a pull model: each application exposes an HTTP endpoint, typically /actuator/prometheus in Spring Boot, that returns metric values, and Prometheus scrapes every target on a fixed interval such as fifteen seconds. The pull model lets Prometheus control the load and immediately detect down targets. Each scrape appends data to a time-series database where every unique combination of a metric name and its labels is a separate series, which is why labels must be kept low-cardinality.
Querying is done with PromQL, a functional language built for this data. The function you reach for most is rate(), which gives the per-second increase of a counter over a window, so an error ratio is the rate of failed requests divided by the rate of total requests. Grafana connects to Prometheus as a data source and turns those queries into live dashboards, and with exemplars it can link a latency spike straight to a sample trace.
The part that matters most is actionable alerting. The single most important tool against false positives is the for duration: the condition must hold continuously for a set time, such as five minutes, before the alert fires, so a one-scrape blip never pages anyone. You alert on symptoms users feel, like a high error ratio or a high p99 latency, not on causes like CPU being busy. In the National Bank of India project, the transfer-service alert fired only when the error ratio for transfers between accounts like Rajesh's SAV2001 and Priya's CUR3001 stayed above one percent for five minutes, so a transient blip was suppressed while a sustained degradation paged the on-call engineer with a runbook link.
Remember one sentence: Prometheus pulls metrics on a schedule, PromQL queries them, Grafana visualizes them, and good alerts fire on sustained user-facing symptoms, never on momentary blips.
---