Definition
- eBPF (extended Berkeley Packet Filter) is a Linux kernel technology that lets you attach small, sandboxed, verified programs to kernel hooks without modifying application code
- The eBPF verifier checks every program before execution, guaranteeing it cannot crash the kernel, loop forever, or access unsafe memory
- eBPF-based observability surfaces kernel-level signals: TCP retransmits, packet drops, run-queue/scheduler latency, disk IO wait, and page faults
- You reach for it beyond APM when a latency spike or pod restart has no visible cause at the application or JVM level
- Tools include bpftrace/BCC for ad-hoc tracing, Pixie for Kubernetes, and Cilium for network observability
Normal application monitoring (APM) is like watching the clerks at the National Bank of India: you see how fast each clerk serves Rajesh and Priya. But sometimes the clerk is fast and correct, yet customers still wait, because the problem is beneath the clerk: the building's elevators are slow, the phone lines keep dropping, or the mail room is jammed. eBPF is like installing silent sensors in the building's infrastructure itself: the elevators, the phone exchange, the mail room, without disturbing a single clerk.
Understanding the Concepts
What is eBPF and how does it work?
eBPF lets you attach tiny programs to Linux kernel hook points: system call entry/exit, network packet events, scheduler context switches, and block-device IO completions. The programs run in kernel space but are first verified to be safe.
# eBPF execution model User-space tool Kernel ┌──────────────┐ ┌────────────────────────────┐ │ bpftrace │───▶│ eBPF Verifier │ │ script │ │ (checks safety) │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ eBPF Program attached to: │ │ │ │ - syscall hooks │ │ │ │ - network hooks (XDP, tc) │ │ │ │ - scheduler hooks │ │ │ │ - disk IO hooks │ │ │ │ │ │ │ │◀───│ eBPF Maps (results) │ └──────────────┘ └────────────────────────────┘
The eBPF verifier is what makes this safe for production: it guarantees the program cannot crash the kernel, cannot loop forever, and cannot access memory it should not.
What signals does eBPF expose that APM cannot?
APM, metrics, and traces see the application and the JVM. eBPF sees everything below that layer.
| Signal | What It Reveals | Why APM Misses It |
|---|---|---|
| TCP retransmits | Packets dropped and re-sent | No application span for network retries |
| Packet drops | Network congestion | Invisible to application code |
| Run-queue latency | Process waiting for CPU | Appears as "slow" but process was not running |
| Disk IO wait | Blocked on slow storage | Not captured by application timers |
| Page faults | Memory pressure, swapping | Invisible to JVM heap metrics |
| Syscall latency | Slow kernel operations | Not tracked by APM spans |
# bpftrace: trace TCP retransmits in real time
bpftrace -e 'kprobe:tcp_retransmit_skb {
printf("Retransmit: %s:%d -> %s:%d\n",
ntop(((struct sock *)arg0)->sk_rcv_saddr),
((struct sock *)arg0)->sk_num,
ntop(((struct sock *)arg0)->sk_daddr),
ntohs(((struct sock *)arg0)->sk_dport));
}'
# Output:
Retransmit: 10.0.1.5:8080 -> 10.0.2.3:45678
Retransmit: 10.0.1.5:8080 -> 10.0.2.3:45678
Retransmit: 10.0.1.5:8080 -> 10.0.2.3:45680
# Packets between transfer-service and account-service are being retransmitted!
When do you reach for eBPF beyond APM?
The trigger is a symptom with no application-level cause. The application looks healthy, the JVM is fine, traces show time spent "somewhere" that no span captures.
# Scenario at NBI:
# APM shows: transfer-service call to account-service took 800ms
# Account-service APM shows: it processed the request in 20ms
# 780ms vanished between the two services -- where?
# APM blind spot: the 780ms was in the network (TCP retransmits)
# and the scheduler (run-queue latency on a noisy-neighbour node)
# eBPF investigation:
# 1. Trace TCP retransmits between the two pods
# 2. Measure scheduler run-queue latency on the node
# Both reveal causes invisible to APM
What tools are available for eBPF observability?
The tooling has matured from low-level scripts to production-ready platforms.
# bpftrace: ad-hoc one-liners for investigation
# Distribution of scheduler run-queue latency (microseconds)
bpftrace -e 'tracepoint:sched:sched_switch {
@[comm] = hist(curtask->se.statistics.wait_sum / 1000);
}'
# BCC tools: pre-built scripts for common investigations
# Trace disk IO latency distribution
biolatency-bpfcc
# Trace file open syscalls slower than 1ms
opensnoop-bpfcc --duration 1
# Pixie: auto-observability for Kubernetes, no code changes
# Install Pixie on the cluster:
# px deploy
# Query service-to-service latency without any instrumentation:
# px run px/service_stats
# Shows: transfer-service -> account-service: p50=20ms, p99=800ms
How is eBPF different from APM? (comparison table)
| Aspect | APM (metrics/traces) | eBPF |
|---|---|---|
| Visibility | Application + JVM | Kernel + network + scheduler |
| Code changes | SDK or agent required | None |
| Overhead | Moderate (instrumentation) | Very low (kernel-level) |
| Use case | Primary monitoring | Escalation when APM is blind |
| Examples | Micrometer, OpenTelemetry | bpftrace, Pixie, Cilium |
| Limitation | Cannot see below the app | Kernel-version and privilege sensitive |
What are common mistakes with eBPF?
The most common mistake is treating eBPF as a replacement for the three pillars instead of an escalation tool.
# WRONG approach:
# "Let's use eBPF for everything and skip Prometheus and Jaeger"
# eBPF is powerful but does not replace structured metrics, logs, and traces
# CORRECT approach:
# 1. Start with metrics, logs, and traces (the three pillars)
# 2. When the pillars point "below the application" without a cause:
# 3. Drop down to eBPF to investigate kernel-level signals
# 4. Once the cause is found, add a permanent metric or alert for it
What should a senior engineer know?
eBPF is kernel-version and privilege sensitive. In production, you use managed tools like Pixie or Cilium with Hubble rather than hand-loading raw BPF programs. Cilium uses eBPF for networking, security, and network observability, giving you a map of actual pod-to-pod traffic and drops without any application instrumentation.
# Cilium Hubble: observe network flows between pods
hubble observe --from-pod transfer-service --to-pod account-service
# Output:
FLOW VERDICT SOURCE DESTINATION SUMMARY
--> ALLOWED transfer-service account-service TCP SYN
--> DROPPED transfer-service account-service TCP RST # packet drop!
What are common interview follow-up questions?
- Is eBPF safe for production? Yes, the verifier guarantees safety. Managed tools like Pixie and Cilium are production-grade.
- Can eBPF see inside the JVM? Not directly. eBPF sees syscalls and kernel events. For JVM internals (heap, GC), you still need JMX or Micrometer.
- What Linux kernel version is needed? eBPF matured in kernel 4.x and 5.x. Most features require kernel 4.18+.
- How does eBPF compare to strace? strace uses ptrace which is slow and intrusive. eBPF hooks into the kernel with minimal overhead.
Visual Overview
APM vs eBPF Visibility
┌──────────────────────────────────────────┐
│ APPLICATION LAYER (APM sees this) │
│ Request rate, Error rate, Latency │
│ JVM heap, GC pauses, Thread count │
│ Database queries, HTTP calls │
├──────────────────────────────────────────┤
│ KERNEL LAYER (eBPF sees this) │
│ TCP retransmits, Packet drops │
│ Scheduler run-queue latency │
│ Disk IO wait, Page faults │
│ Syscall latency │
└──────────────────────────────────────────┘
Investigation Escalation Path
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Metrics │───▶│ Traces │───▶│ Logs │
│ (alert) │ │ (locate) │ │ (why) │
└──────────┘ └──────────┘ └────┬─────┘
│
No app cause?│
▼
┌──────────┐
│ eBPF │
│ (kernel) │
└──────────┘
Key Takeaway
eBPF is a Linux kernel technology that lets you attach small, sandboxed, verified programs to kernel hook points like system calls, network events, scheduler context switches, and disk IO, so you can observe what the operating system is doing with no changes to application code. The eBPF verifier checks every program before execution, guaranteeing safety for production use. eBPF-based observability surfaces signals like TCP retransmits, packet drops, run-queue latency, disk IO wait, and page faults.
The reason it matters is that normal APM only sees the application and the JVM. When your code is fast, your JVM is healthy, and the trace shows time spent somewhere that no span captures, the cause is in the kernel, the network stack, or the scheduler, and APM cannot see it. That is when you reach for eBPF.
In the National Bank of India project, the transfer service had intermittent latency spikes with perfect APM: zero errors, healthy heap, quiet garbage collection. A trace showed the account-service call took 800ms while the account service itself reported only 20ms, so 780ms vanished between the two services. Dropping to eBPF, the team traced TCP retransmits and found a storm of retransmits inflating latency, and measured run-queue latency showing the node was CPU-starved by a noisy neighbour pod. Both causes were below the application and invisible to APM.
Remember one sentence: eBPF is an escalation, not a replacement: start with the three pillars, and when they point below the application without pinpointing the cause, eBPF lets you see into the kernel with zero code changes.
---