The perf command in Linux is a powerful tool for performance analysis, but its data files are often a black box.
Let’s see perf in action. Imagine you’re tracking CPU cache misses on a specific process.
perf record -p $(pgrep my_application) sleep 10
This command starts recording performance data for the process with PID $(pgrep my_application) for 10 seconds. The output is a perf.data file in the current directory.
Now, to read that data:
perf script -i perf.data
This will dump a human-readable trace of events to standard output. You’ll see timestamps, event types (like cpu-cycles, cache-misses), and associated data.
The core idea behind perf data files is event-based sampling. When the kernel’s performance monitoring unit (PMU) detects a specific event (like a cache miss or a certain number of CPU cycles), it triggers an interrupt. perf hooks into this, records the event, and crucially, the program counter (PC) value at the time of the event. This PC value is what allows us to attribute performance issues to specific functions or lines of code.
To analyze this, we often use perf report:
perf report -i perf.data
This presents an interactive TUI (Text User Interface) where you can explore the collected data. It aggregates events by symbol (function names), allowing you to see which functions are responsible for the most CPU cycles, cache misses, or other profiled events. You can drill down into specific symbols to see the distribution of events within them, often down to the assembly instruction level if debug symbols are available.
The data file itself is a binary format, optimized for efficient recording. It contains raw event data, system information (like CPU topology, kernel version), and symbol tables. The perf script and perf report commands parse this binary data, interpret the raw event codes, and resolve addresses to symbol names using the system’s symbol tables.
Here’s a typical workflow:
- Record:
perf record <command_or_process> - Analyze:
perf report(orperf scriptfor raw output)
The real power comes from understanding the types of events you can record. Beyond basic CPU cycles and cache misses, you can profile:
- Hardware Events: These are directly supported by your CPU’s PMU. Examples include instructions retired, branch misses, L1/L2/LLC cache misses, and memory access latency. You can list available hardware events with
perf list. - Software Events: These are generated by the kernel itself, not directly by the CPU hardware. Examples include context switches, page faults, and system calls.
- Tracepoints: These are specific, instrumented points within the kernel code that can be enabled to record detailed information about kernel operations. Think of them as kernel-level debugging hooks.
The perf.data file is essentially a log of these events. The perf record command tells the kernel’s perf_event subsystem to start monitoring specific events for a given process or the entire system. When an event occurs, the kernel generates an perf_event_sample structure, which perf record then writes to the perf.data file. This structure includes the event type, the timestamp, the sample data (like the PC value), and other contextual information.
When you run perf report, it reads this file sequentially. It builds histograms of events, mapping them back to symbols using the symbol information also stored in the file or by looking up symbols in the running kernel and loaded libraries. The interactive report allows you to sort, filter, and navigate this data. For instance, you can filter by specific event types, sort by percentage of total events, and expand symbols to see their constituent functions.
The most surprising thing about perf data files is how much information about the entire system can be captured within them, even when profiling a single process. perf record doesn’t just capture events related to your target process; it often records system-wide events that might be impacting your process’s performance, like heavy I/O from other processes or kernel activity. The perf.data file can contain CPU frequency scaling events, thermal throttling events, and even information about other running processes if system-wide profiling was enabled. This holistic view is crucial for diagnosing subtle performance bottlenecks that aren’t directly caused by your application’s code.
If you’re running perf record as a non-root user, you’ll often encounter permission issues or a limited set of available events. The perf_event_paranoid kernel setting controls what unprivileged users can do. To get the most comprehensive data, especially for hardware-level events, running perf record with sudo is usually necessary.
The next step after analyzing a perf.data file is often to correlate the sampled events with source code for deeper debugging.