Redpanda’s max_bytes configuration isn’t just a soft cap; it’s a hard boundary enforced by the broker that can prevent producers from sending messages larger than the configured limit.
Here’s how to see it in action. Imagine you have a Redpanda cluster running locally, and you’ve set the default max_bytes to 1MB.
# Check current config (example, actual path may vary)
rpk cluster config get max_bytes
# Output might look like:
# max_bytes: 1048576
Now, try to produce a message slightly larger than that limit to a topic.
# Create a dummy file larger than 1MB
dd if=/dev/zero of=large_message.bin bs=1M count=2
# Attempt to produce it (assuming topic 'my-topic' exists)
rpk topic produce my-topic < large_message.bin
You’ll likely see an error message from Redpanda, something like:
Error: Failed to produce message: Broker: Message Too Large
This is the max_bytes limit in action. The broker received the producer’s request, checked the message size against its configured max_bytes, and rejected it because it exceeded the threshold. The producer then fails.
To address this, you need to increase the max_bytes configuration on your Redpanda cluster. This is done via rpk cluster config update.
First, check your current configuration to see what it’s set to:
rpk cluster config get max_bytes
Let’s say it’s currently 1048576 (1MB). If you need to allow messages up to 10MB, you would update it like this:
rpk cluster config update max_bytes 10485760
This command sends an update to the Redpanda cluster’s configuration. Redpanda brokers will then pick up this new configuration value. It’s important to note that this change needs to be applied to all brokers in your cluster for consistent behavior. rpk cluster config update typically handles this across the cluster, but it’s good to be aware.
After the update, Redpanda brokers will now accept messages up to 10MB. Producers attempting to send messages within this new limit will succeed, while messages exceeding it will still be rejected with the same "Message Too Large" error. The broker enforces this limit at the network layer before a message is even written to disk or replicated.
It’s also crucial to understand that this setting is a per-message limit, not a per-batch limit. Producers often batch records together before sending them to the broker. The max_bytes configuration applies to the size of an individual record within that batch. If a producer tries to send a single record that exceeds max_bytes, it will be rejected.
Furthermore, client libraries often have their own configurable message size limits (e.g., max.request.size in Kafka clients). These client-side limits must be at least as large as the broker’s max_bytes setting for messages to be sent successfully. If the client’s max.request.size is smaller than the broker’s max_bytes, the client will reject the message before it even reaches the broker. So, if you increase max_bytes on the broker, you must also ensure your producer clients are configured to send larger messages.
For example, if you’re using a Java Kafka producer, you’d set:
max.request.size=10485760 # 10MB
And for the broker:
rpk cluster config update max_request_size 10485760
While max_bytes is the broker-side limit, max_request_size is the client-side equivalent. They work in tandem.
The max_bytes configuration is stored in Redpanda’s internal configuration store and is replicated across all nodes. When you issue an rpk cluster config update, Redpanda takes that value and applies it to its runtime configuration. The actual value is often read from ZooKeeper or etcd (depending on your Redpanda deployment mode), and then broadcast to all brokers. The brokers then use this value to check incoming messages.
A common mistake is to only update the max_bytes setting on one broker or to forget to update the client-side max.request.size. Both must be aligned for larger messages to flow.
The ultimate consequence of ignoring message size limits is not just producer failures but also potential data loss if producers are not configured with appropriate error handling and retry mechanisms for "Message Too Large" errors. They might simply stop sending data without a clear indication of why, leading to data gaps.
The next thing you’ll likely encounter is dealing with the message.max.bytes setting within topic configurations.