Pulsar’s "max consumers" setting on a topic isn’t about limiting how many clients can read from a topic, but how many can be actively connected and consuming at any given moment.
Let’s see it in action. Imagine a topic named persistent://public/default/my-topic. We want to ensure no more than 10 consumers can be actively reading from it.
Here’s how you’d configure that using pulsar-admin:
pulsar-admin topics set-max-consumers persistent://public/default/my-topic --max 10
Now, if you try to start an 11th consumer on that topic, it will be rejected by the broker. The broker will send a TooManyRequests (HTTP 429) response to the client attempting to connect.
The Problem Pulsar Solves Here
Normally, a topic can have an arbitrary number of consumers. However, in certain scenarios, especially with shared subscriptions, having too many active consumers can lead to:
- Increased Broker Load: Each consumer connection consumes resources (memory, CPU, network sockets) on the broker.
- Message Acknowledgment Challenges: In shared subscriptions, Pulsar needs to track acknowledgments from all consumers to know when a message is fully processed. More consumers mean more state to manage.
- Potential for Over-Subscription: If a single producer is sending data very rapidly, and you have a vast number of consumers, the system might struggle to keep up with delivering messages to all of them, leading to increased latency or dropped messages if not handled carefully.
The max-consumers setting provides a hard, configurable limit to prevent these issues by capping the number of active subscribers.
How it Works Internally
When a consumer attempts to connect to a topic with a specific subscription type (like Shared or Key_Shared), the broker handling the connection checks the current count of active consumers for that subscription.
- Connection Attempt: A client (consumer) tries to connect to a broker for a specific topic and subscription.
- Broker Check: The broker looks up the topic and subscription. It maintains an internal counter for active consumers associated with that subscription.
- Limit Exceeded? If the current count is less than the configured
max-consumersfor that topic, the connection is allowed, and the counter is incremented. - Limit Reached: If the current count is already equal to or greater than
max-consumers, the broker immediately rejects the connection attempt. The response is typically an HTTP 429 Too Many Requests, and the client library will likely report a connection error. - Disconnection: When a consumer disconnects gracefully or is forcefully disconnected, the broker decrements the active consumer counter for that subscription.
This check happens at the broker level during the initial connection handshake. It’s not a per-topic setting in the sense that it applies to all subscriptions on the topic; rather, you set it on the topic, and it acts as a global cap for all subscriptions of that topic. If you have multiple subscriptions on the same topic, the max-consumers limit applies to the sum of active consumers across all those subscriptions.
Configuration Details
- Scope: The
max-consumerssetting is applied to a specific topic. - Replication: This setting is managed by ZooKeeper (or the configured metadata store) and will be replicated across all brokers.
- Default: If not set, there is effectively no limit imposed by Pulsar itself (though system resource limits will always apply).
- Dynamic: You can change this value dynamically using
pulsar-adminwithout restarting brokers or clients. Clients will be disconnected if the limit is lowered below the current active consumer count.
The One Thing Most People Don’t Know
The max-consumers limit is enforced per topic, not per subscription. This means if you have a topic my-topic with max-consumers set to 5, and you have two subscriptions, sub1 and sub2, the total number of active consumers across both sub1 and sub2 cannot exceed 5. If sub1 has 3 active consumers and sub2 has 3 active consumers, the next consumer trying to connect to either sub1 or sub2 will be rejected.
Next Steps
Once you’ve mastered topic limits, you’ll want to explore how Pulsar handles message redelivery and backlog quotas to manage data lifecycle and consumer health.