Redpanda’s Seastar memory allocator is designed to give each CPU core its own dedicated memory pool, preventing contention and improving performance.
Here’s how it looks in action:
Imagine you have a Redpanda cluster with 4 cores per node. When Redpanda starts, Seastar will request a significant chunk of memory from the OS, but it won’t just be one big pool. Instead, it will carve out separate memory arenas, one for each core.
Let’s say you have 10GB of RAM allocated to Redpanda. Seastar might request this from the OS, but internally, it’s managing it like this:
- Core 0: Gets its own memory pool (e.g., 2.5GB)
- Core 1: Gets its own memory pool (e.g., 2.5GB)
- Core 2: Gets its own memory pool (e.g., 2.5GB)
- Core 3: Gets its own memory pool (e.g., 2.5GB)
When a thread running on Core 0 needs to allocate memory, it only tries to allocate from Core 0’s pool. This is key. It avoids the need for locks that would be required if all cores were contending for a single, shared memory pool.
The problem Seastar solves is the classic multi-threaded memory allocation bottleneck. In traditional allocators, when multiple threads (each potentially on a different CPU core) try to allocate or free memory simultaneously, they often have to acquire a global lock. This lock serializes access, meaning only one thread can be in the allocator at a time, even if there are plenty of CPU cores available. This becomes a massive performance killer, especially under high load. By giving each core its own allocator instance, Seastar eliminates this contention.
The primary lever you control for this is the total memory allocated to Redpanda and, implicitly, how many cores are available to Redpanda. Redpanda’s seastar.memory.max-mb configuration parameter dictates the total memory Seastar can use. If you have 16 cores and set seastar.memory.max-mb: 64000 (64GB), Seastar will divide that 64GB across the 16 cores, giving each core roughly 4GB of dedicated memory.
Here’s a snippet from a redpanda.yaml that influences this:
# ... other configurations ...
seastar.memory.max-mb: 32000 # Allocate 32GB total for Seastar
# ... other configurations ...
If you were to change this to seastar.memory.max-mb: 16000 (16GB) on a 16-core machine, each core would now have only 1GB of its dedicated pool, potentially leading to increased allocation failures or more frequent garbage collection within Seastar’s allocator if it runs out of space in a core’s pool.
The most surprising aspect of Seastar’s per-core allocation is how it handles "out of memory" situations within a single core’s pool. If Core 0’s pool is exhausted, it doesn’t immediately fail. Instead, it will attempt to "steal" memory from another core’s pool. This stealing operation is more expensive than a local allocation because it requires cross-core communication and potential synchronization, but it’s designed to be a fallback, not the primary mechanism. The goal is to keep allocations local to the core as much as possible.
The next problem you’ll encounter is understanding how Seastar’s internal garbage collection mechanisms interact with these per-core memory pools when memory pressure arises.