Redpanda SMP Core Pinning dedicates specific CPU cores to Redpanda threads, preventing the operating system’s scheduler from moving them around, which dramatically reduces cache misses and context switching overhead.
Let’s see it in action. Imagine a Redpanda cluster with 16 CPU cores. We want to dedicate cores 0-7 to Redpanda’s event loops and I/O threads, and cores 8-15 to the OS and other applications.
First, we need to identify the threads Redpanda uses. When Redpanda starts, it typically creates several threads for its core operations. You can often see these threads by using htop or top and filtering by the redpanda process. You’ll see threads like redpanda: rpc, redpanda: raft, redpanda: io, redpanda: wasm, etc.
To pin these threads, we’ll use the taskset command. Let’s say your Redpanda process ID (PID) is 12345.
We can start Redpanda with core pinning directly using the redpanda binary’s --smp-threads and --smp-affinity flags. This is the most robust way.
For example, to pin Redpanda to cores 0-7, you would start it like this:
redpanda --smp-threads 8 --smp-affinity 0x0101010101010101
Here’s the breakdown:
--smp-threads 8: Tells Redpanda to utilize 8 threads for its SMP operations.--smp-affinity 0x0101010101010101: This is the crucial part. It’s a bitmask where each bit corresponds to a CPU core. A1means the thread can run on that core, and a0means it cannot. The value0x0101010101010101in hexadecimal represents the binary mask00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001. For a 64-bit system, this mask means cores 0, 8, 16, 24, 32, 40, 48, and 56 are enabled. This isn’t what we want for contiguous cores 0-7.
A more practical example for cores 0-7 on a system with at least 8 cores would be:
redpanda --smp-threads 8 --smp-affinity 0xFF
0xFFin hex is11111111in binary, enabling cores 0 through 7.
If you need to pin Redpanda to a different set of cores, say 4-11, you’d calculate the mask accordingly. For cores 4-11 (8 cores total), the mask would be 0x1FF8.
- Binary:
00000001 11111111 1000 - Cores enabled: 4, 5, 6, 7, 8, 9, 10, 11.
You can also achieve this by modifying the redpanda.yaml configuration file:
# redpanda.yaml
smp:
threads: 8
affinity: "0xFF" # For cores 0-7
After starting Redpanda with these configurations, you can verify the pinning using htop or ps. In htop, select the Redpanda process, press F2 (Setup), go to Columns, and add "CPU" and "CPU%". You’ll see that the Redpanda threads are exclusively running on the specified cores.
The core idea is that when a CPU core is dedicated, its L1, L2, and L3 caches are populated with data relevant to Redpanda’s operations. When the OS scheduler is allowed to move threads freely, a thread might be scheduled on a core whose cache is filled with data from a completely different process. This forces the new thread to fetch its data from slower main memory, causing a cache miss and a significant performance hit. Core pinning ensures that the cache remains "warm" with Redpanda’s data, leading to drastically fewer cache misses and faster thread execution. Context switching also becomes less frequent, as the OS doesn’t need to preempt and reschedule these pinned threads as often.
Beyond the explicit --smp-affinity flag, Redpanda’s internal thread management also plays a role. The smp.threads setting dictates how many general-purpose worker threads Redpanda creates, and the smp.affinity mask is applied to all of these threads. However, Redpanda uses specific thread pools for different tasks (RPC, raft, I/O, etc.). The affinity mask applies to all of them.
A common misconception is that setting a high number for --smp-threads automatically maps to available cores. This is not true; without an affinity mask, Redpanda threads will be subject to the OS scheduler’s whims. Another pitfall is miscalculating the affinity mask, especially on systems with many cores, leading to threads being pinned to unexpected or non-contiguous cores, negating the intended performance gains.
The next challenge you’ll face is understanding how network interface card (NIC) interrupts interact with your core-pinned Redpanda threads.