Redpanda can handle massive throughput, but its limits are more about network and CPU than disk I/O.
Let’s see Redpanda chug. We’ll use the rpk bench tool, which is built right into Redpanda. First, make sure you have a Redpanda cluster running. For this, a single-node cluster is fine, but a multi-node cluster will give you a more realistic view of its scaling.
Here’s how to generate some load. We’ll send messages to a topic called load_test_topic.
rpk topic create load_test_topic
rpk bench producer --brokers localhost:9092 --topic load_test_topic --messages 1000000 --rate 100000
This command tells rpk to:
- Connect to
localhost:9092(your Redpanda broker). - Send messages to the
load_test_topic. - Send a total of 1,000,000 messages.
- Attempt to send them at a rate of 100,000 messages per second.
The --rate flag is where the magic happens. Redpanda’s internal scheduler will try to hit this target. You’ll see output like this:
...
Sent 1000000 messages in 10.00s (100000.00 msg/sec)
...
Now, let’s measure consumption. We’ll run a consumer that reads from the same topic.
rpk bench consumer --brokers localhost:9092 --topic load_test_topic --messages 1000000 --rate 100000
This consumer will try to read messages at the same rate. The output will show how many messages it could process and how long it took. If the producer rate was too high for the consumer (or Redpanda itself), you’ll see a lower measured rate.
The key to understanding Redpanda’s performance lies in its architecture. It’s a single-binary Kafka-compatible streaming platform built on the Seastar C++ framework. Seastar is designed for high-performance, asynchronous I/O, using a work-stealing scheduler across CPU cores. This means Redpanda can utilize multiple CPU cores very efficiently for both producing and consuming messages, as well as for its internal raft replication.
When you stress test, you’re primarily testing how many network packets and CPU cycles Redpanda can push through. Disk I/O is often not the bottleneck because Redpanda uses a log-structured merge-tree (LSM tree) for its storage. Data is written sequentially to memory buffers and then flushed to disk in larger, optimized chunks. The latency for writes is very low, and reads can be very fast if data is cached.
The --rate parameter in rpk bench is not a hard limit enforced by the broker. It’s a target for the client. If the broker can’t keep up, the client will experience higher latency and potentially drop below the target rate. Redpanda’s internal metrics, accessible via rpk metrics dump, will tell you what’s happening on the server side: look for kafka_server_request_handler_time_ms and kafka_server_bytes_in_per_sec. High values here indicate the broker is struggling to process requests.
The actual limit you hit will depend on several factors:
- CPU Cores: Redpanda scales very well with CPU. More cores mean more capacity to handle requests and replication.
- Network Bandwidth: Even with fast CPUs, if your network can’t keep up, you’ll hit a wall.
- Client/Producer Performance: The
rpk benchtool is a good baseline, but real-world applications might have less efficient serialization or network stacks. - Topic Configuration: Number of partitions, replication factor, and message size all influence throughput.
- Redpanda Configuration: Settings like
max_transaction_timeout_msormax_request_sizecan indirectly affect throughput under extreme load.
One subtle point is how Redpanda’s event-driven model interacts with network buffers. When the system is saturated, network buffers can fill up. TCP congestion control will then start to back off, reducing the effective data rate. This isn’t a Redpanda-specific issue, but rather a characteristic of high-throughput network applications. The system tries to send data faster than the network or CPU can process it, and TCP, in its wisdom, slows things down to prevent packet loss, which can mask the true CPU/network capacity of the Redpanda nodes themselves.
The next thing you’ll likely want to explore is how Redpanda behaves under sustained load over longer periods, and how its distributed consensus (Raft) impacts write performance when you increase the replication factor.