Pulsar clients often feel like black boxes, but their performance is entirely tunable by understanding a few core concepts.
Let’s look at a Pulsar producer. Imagine you’re sending a flood of messages into a river. The producer is the person doing the throwing. By default, it’s not going to throw them as fast as it possibly can. It’s going to try to be polite and efficient.
Here’s a producer in action, configured for higher throughput:
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.build();
Producer<byte[]> producer = client.newProducer()
.topic("persistent://public/default/my-topic")
.enableBatching(true)
.batchingMaxPublishDelay(10, TimeUnit.MILLISECONDS)
.batchingMaxMessages(1000)
.maxPendingMessages(5000)
.sendTimeout(30, TimeUnit.SECONDS)
.create();
// Send some messages
for (int i = 0; i < 10000; i++) {
producer.sendAsync(("message-" + i).getBytes());
}
producer.close();
client.close();
This producer is configured to batch messages (enableBatching(true)). Instead of sending each message individually, it waits up to 10 milliseconds or until it has 1000 messages, whichever comes first, and sends them as a single batch. This drastically reduces the network overhead per message. maxPendingMessages(5000) means the producer will buffer up to 5000 messages if the broker is slow to acknowledge them, preventing it from blocking your application thread too aggressively. sendTimeout(30, TimeUnit.SECONDS) is a safety net, ensuring that a stuck send operation doesn’t hang indefinitely.
Now, let’s flip it to a consumer. A consumer is the person downstream trying to catch those messages.
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.build();
Consumer<byte[]> consumer = client.newConsumer()
.topic("persistent://public/default/my-topic")
.subscriptionName("my-subscription")
.receiverQueueSize(1000)
.ackTimeout(1, TimeUnit.MINUTES)
.subscribe();
while (true) {
Message<byte[]> msg = consumer.receive();
try {
// Process the message
System.out.println("Received message: " + new String(msg.getData()));
consumer.acknowledge(msg);
} catch (Exception e) {
// Handle processing error, negative acknowledge to retry
consumer.negativeAcknowledge(msg);
}
}
consumer.close();
client.close();
The receiverQueueSize(1000) is crucial. This is the internal buffer on the client side for messages pulled from the broker but not yet processed and acknowledged. A larger queue means the consumer can handle bursts of messages from the broker without blocking the receiving thread, allowing your processing logic to run at its own pace. ackTimeout(1, TimeUnit.MINUTES) is the duration the broker will wait for an acknowledgment before redelivering the message. If your processing takes longer than this, messages will be resent, potentially causing duplicates if not handled idempotently.
The total number of pending messages in the system, from producer to broker to consumer, is a critical factor in throughput and latency. Each component has its own buffer, and when these buffers fill up, backpressure is applied. For producers, maxPendingMessages is the primary lever. For consumers, receiverQueueSize controls the client-side buffer. The broker itself also has internal limits.
The batchingMaxPublishDelay is one of the most impactful settings for producer throughput. Setting it too low, like 1ms, will result in many small batches, increasing overhead. Setting it too high might increase latency if you have a steady stream of messages but few of them. The optimal value depends on your message arrival rate and latency requirements.
A common misconception is that a larger receiverQueueSize always leads to higher consumer throughput. While it can smooth out processing, if the queue becomes excessively large, it can mask underlying processing bottlenecks and increase end-to-end latency. The broker might even start dropping messages if its own internal buffers are full and it can’t send them to the consumer fast enough, even if the consumer has a large receiverQueueSize.
The sendTimeout on the producer is not about how fast you send, but how long you’re willing to wait for the broker to confirm receipt of a message (or a batch). If the broker is overloaded or unreachable, this timeout prevents your application from hanging.
When tuning, remember that producer and consumer configurations are often interdependent. A very fast producer can overwhelm a slow consumer, and vice-versa. Monitoring the Pulsar broker’s internal metrics (like msgRateIn, msgRateOut, unackedMsgs) and client-side metrics (like getSendQueueSize for producers, and getQueueSize for consumers) is essential for identifying where the bottleneck lies.
The next logical step after tuning individual clients is understanding how Pulsar’s tiered storage can offload older data to cheaper storage, impacting read performance for historical data.