Linux perf and eBPF are both powerful tracing and profiling tools, but they operate on fundamentally different principles, making one a better choice than the other depending on your specific needs. The most surprising truth is that eBPF isn’t just a "better perf"; it’s a paradigm shift that allows you to run sandboxed C programs directly in the Linux kernel, offering unprecedented flexibility and safety.
Let’s see perf in action. Imagine you want to profile CPU usage for a specific process.
# Get process ID for 'my_application'
pgrep my_application
# Assuming PID is 12345
perf top -p 12345
This command will show you a real-time, interactive view of which functions are consuming the most CPU time within process 12345. perf excels at hardware performance counters and kernel tracepoints/kprobes, giving you insights into CPU cycles, cache misses, branch predictions, and function call distributions. It’s excellent for understanding what is happening at a low level, especially when performance bottlenecks are suspected.
Now, consider eBPF. It’s not about sampling hardware events directly in the same way perf does. Instead, you write small C programs that can be attached to various kernel hooks (like network packet reception, syscall entry/exit, or function calls). The kernel verifies these programs for safety (e.g., preventing infinite loops or kernel crashes) and then executes them in a sandboxed environment.
Here’s a conceptual eBPF program that counts incoming TCP packets on a specific port:
// This is a simplified representation, actual eBPF requires specific headers and macros.
#include <uapi/linux/bpf.h>
#include <uapi/linux/if_ether.h>
#include <uapi/linux/ip.h>
#include <uapi/linux/tcp.h>
// A map to store our counts, keyed by port number
struct bpf_map_def SEC("maps") port_counts = {
.type = BPF_MAP_TYPE_HASH,
.key_size = sizeof(u16),
.value_size = sizeof(u64),
.max_entries = 256,
};
SEC("xdp")
int count_tcp_packets(struct xdp_md *ctx) {
void *data_end = (void *)(long)ctx->data_end;
void *data = (void *)(long)ctx->data;
struct ethhdr *eth = data;
if ((void *)(eth + 1) > data_end) return XDP_PASS;
if (eth->h_proto == htons(ETH_P_IP)) {
struct iphdr *iph = data + sizeof(struct ethhdr);
if ((void *)(iph + 1) > data_end) return XDP_PASS;
if (iph->protocol == IPPROTO_TCP) {
struct tcphdr *tcph = (void *)iph + iph->ihl * 4;
if ((void *)(tcph + 1) > data_end) return XDP_PASS;
u16 sport = tcph->source; // Network byte order
u64 *count = bpf_map_lookup_elem(&port_counts, &sport);
if (count) {
(*count)++;
} else {
u64 initial_count = 1;
bpf_map_update_elem(&port_counts, &sport, &initial_count, BPF_ANY);
}
}
}
return XDP_PASS; // Pass the packet along
}
char _license[] SEC("license") = "GPL";
This eBPF program, once compiled and loaded, would attach to the XDP (eXpress Data Path) hook on a network interface. When a packet arrives, this program runs, inspects it, and increments a counter for the source TCP port in a kernel map. You can then read this map from userspace to see the counts.
The core problem eBPF solves is safe, in-kernel programmability without modifying kernel source code or loading kernel modules. perf is primarily a profiler and tracer that leverages existing kernel instrumentation. eBPF is a programmable tracing and networking framework.
perf is great for understanding CPU-bound issues, cache behavior, instruction-level bottlenecks, and general kernel event tracing. It’s often the first tool to reach for when you suspect a performance problem. eBPF, on the other hand, is ideal for custom network monitoring, security auditing, dynamic tracing of any kernel or userspace function, and building complex observability tools. It allows you to write logic that runs where the event happens, reducing overhead and enabling more sophisticated analysis.
A key difference is how they handle data. perf typically collects raw event data and processes it in userspace (e.g., perf report). eBPF allows you to do much of the aggregation and filtering within the kernel using BPF maps, sending only the summarized results to userspace. This significantly reduces the amount of data that needs to be transferred and processed, leading to lower overhead for complex tracing scenarios. For instance, instead of collecting every single network packet header, an eBPF program can just count specific types of packets and store the aggregate count in a map.
The one thing most people don’t know is that eBPF programs are not just event-driven; they can also maintain state across events using BPF maps. These maps are shared data structures that live in the kernel and can be accessed by both eBPF programs and userspace applications. This statefulness is what allows eBPF to build sophisticated applications like distributed tracing, network traffic analysis, and security policy enforcement directly within the kernel, without needing to constantly poll or rely on external state management.
While perf is excellent for understanding existing performance characteristics, eBPF opens the door to dynamically shaping and observing system behavior in ways previously thought impossible without kernel module development.