The Linux perf tool, often seen as a low-level CPU profiler, is surprisingly powerful for digging into network stack performance issues.
Let’s see perf in action, profiling a simple ping to a local host. We’ll use perf record to capture events and then perf report to analyze them.
sudo perf record -e net:net_dev_queue -e net:net_rx_action -e net:net_tx_complete -a -- sleep 5
sudo perf report
This command records network-related events for 5 seconds. The net:net_dev_queue, net:net_rx_action, and net:net_tx_complete events give us visibility into packets being queued for transmission, the network device receiving packets, and the completion of transmissions, respectively. The -a flag samples across all CPUs.
The perf report output will show you a breakdown of where the CPU time was spent within the kernel’s network stack. You’ll see functions like __netif_tx_schedule, dev_queue_xmit, rps_receive_skb, and napi_gro_receive. Understanding these functions is key to diagnosing bottlenecks.
The core problem perf helps solve here is attributing network I/O latency or throughput limitations to specific kernel functions or hardware interactions. Instead of guessing, you can see exactly which part of the network stack is consuming the most CPU time or is being invoked most frequently under load.
Let’s break down the mental model:
- Event Tracing:
perfhooks into kernel tracepoints. These are pre-defined points in the kernel code where specific events occur. For networking, these include packet queuing, reception, transmission, and interrupt handling. - Sampling:
perfsamples these events at a high frequency. When an event occurs,perfrecords the current program counter and associated context. Over time, this builds a statistical profile of where the CPU is spending its time. - Analysis:
perf reportaggregates these samples. It shows which functions were active when the samples were taken, allowing you to identify the hottest spots. For network performance, this often points to interrupt handlers, packet processing loops (like NAPI), and queue management functions.
The levers you control are primarily the events you choose to trace and the duration of the trace. You can also filter by process ID (-p <pid>) to focus on a specific application’s network activity.
For example, to focus on the ping process:
# Find the PID of ping
pgrep ping
# Assuming PID is 12345
sudo perf record -e net:net_dev_queue -e net:net_rx_action -e net:net_tx_complete -p 12345 -- sleep 5
sudo perf report
This narrows the scope to just the network events generated by the ping process itself.
The most surprising thing about perf for network profiling is its ability to reveal software bottlenecks even when you suspect hardware issues. You might see high CPU usage in functions like napi_gro_receive or rps_receive_skb, indicating that the CPU is struggling to process incoming packets efficiently, even if the network card itself is capable. This often leads to tuning parameters like interrupt coalescing, Receive Side Scaling (RSS), or Generic Receive Offload (GRO).
Consider a scenario where perf report shows significant time spent in net_rx_action. This function is the heart of the Networked Interrupt-driven I/O (NAPI) polling mechanism. If net_rx_action is consistently high, it means the kernel is spending a lot of CPU cycles polling for and processing incoming packets, rather than being interrupted. This can be a sign that the network card is generating too many interrupts or that the processing per packet is too high. You might then investigate NAPI’s weight parameter or the interrupt coalescing settings on your network interface card (ethtool -c <interface>).
When you move beyond basic network events, you can also use perf to profile specific kernel modules or even user-space network libraries by selecting different tracepoints or kprobes. This allows for extremely granular analysis of where your application’s network I/O is actually happening.
The next logical step after profiling network stack performance is often to investigate the underlying hardware and driver interactions using tools like ethtool and ip link.