Performance profiling isn’t about finding the slowest part of your application; it’s about finding the part that’s wasting the most CPU cycles or memory, which might not be the part that feels slowest.
Let’s say you’ve got a web service, and users are complaining about latency. Your first instinct is to throw strace at it and see what system calls are taking forever. But strace is like looking at the engine from across the street. You see activity, but not the why.
Here’s a typical workflow to actually diagnose and fix performance issues:
1. Establish a Baseline
Before you change anything, you need to know where you are. Run your application under a realistic load. If it’s a web service, use wrk or ab to hit it with a few hundred concurrent users. If it’s a batch job, run it with a representative dataset.
# Example for a web service
wrk -t4 -c100 -d10s http://localhost:8080/api/users
Record key metrics: average response time, p95/p99 response times, CPU utilization, memory usage, and garbage collection activity (if applicable). This is your baseline. You’ll compare your "fixed" results against this.
2. Identify the Hotspots (CPU)
For CPU-bound issues, you need a profiler that samples your application’s execution stack. perf on Linux is a fantastic, low-overhead tool.
# Install if you don't have it
# sudo apt-get install linux-perf
# Run perf, sampling for 30 seconds
sudo perf record -g -F 99 -- sleep 30
# Analyze the results
sudo perf report
perf report presents a TUI where you can navigate through functions. Look for functions consuming a high percentage of samples. The -g flag includes call graph information, which is crucial. You’re not just looking for a function like process_request, but for the specific line of code within process_request that’s being called repeatedly.
3. Identify the Hotspots (Memory)
Memory leaks or excessive allocations can cripple performance. For Go, pprof is your best friend. For Java, jmap and jhat or commercial tools.
// In your Go application, expose pprof endpoints
import _ "net/http/pprof"
// Start an HTTP server that includes pprof
go func() {
log.Println(http.ListenAndServe("localhost:6060", nil))
}()
Then, use go tool pprof to fetch and analyze the heap profile:
# Get the heap profile from a running application
go tool pprof http://localhost:6060/debug/pprof/heap
# Once inside the pprof interactive prompt:
# top - showing the top functions by memory allocation
# list <function_name> - show lines of code for a specific function
Look for functions that are allocating a disproportionate amount of memory, especially those that are allocating repeatedly and not freeing.
4. Deep Dive with Tracing
Profilers give you a snapshot. Tracers show you the flow of execution over time. eBPF tools like bpftrace are incredibly powerful for this.
# Example: Trace all read() syscalls and print their duration
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read /pid == $target_pid/ { @start[tid] = nsecs; } kretprobe:sys_read /pid == $target_pid && @start[tid] != 0/ { $duration = nsecs - @start[tid]; printf("read() took %d ns\n", $duration); delete(@start[tid]); }' -p <your_process_id>
This lets you see exactly how long individual operations are taking, not just aggregated percentages. You might discover that a seemingly fast function is actually being called millions of times, and each call has a small but cumulative overhead.
5. The "Why": Cache Invalidation and Data Structures
Often, the performance bottleneck isn’t a slow algorithm, but how you’re using data. A common culprit is repeated, expensive computations that could be cached. For example, if you’re repeatedly fetching and parsing the same configuration file on every request, that’s a prime candidate for caching.
Consider this Go snippet:
// Bad: Re-parses config on every call
func HandleRequest(req *http.Request) {
config := parseConfig("/etc/myapp/config.json") // Expensive!
// ... use config ...
}
// Good: Parses config once and caches it
var globalConfig *Config
func init() {
globalConfig = parseConfig("/etc/myapp/config.json")
}
func HandleRequest(req *http.Request) {
// ... use globalConfig ...
}
Or, it could be an inefficient data structure. If you’re doing frequent lookups in a slice where a map would be O(1) instead of O(n), that’s a huge win.
6. The "Why": Concurrency and Contention
If your application is multi-threaded or uses goroutines, contention on locks or shared resources can kill performance. pprof in Go can help visualize lock contention. For other languages, you might need specific profilers or use tracing to identify mutex.Lock() or similar calls that are blocking for extended periods.
# In pprof for Go, after fetching a profile:
# mutex - shows goroutines blocked on mutexes
If you see high contention, you might need to:
- Reduce the critical section’s scope.
- Use finer-grained locks.
- Switch to lock-free data structures if possible.
- Use concurrent data structures provided by libraries.
7. Re-evaluate and Iterate
After making a change, go back to Step 1. Establish a new baseline and compare it to your previous one. Did the change improve things? Did it shift the bottleneck elsewhere? Performance tuning is an iterative process. You might find that fixing one bottleneck reveals another, previously masked by the first.
The next error you’ll hit is a resource temporarily unavailable when trying to bind to a port that’s already in use after your service starts up much faster.