Pulsar subscriptions don’t just deliver messages; they govern how those messages are consumed, and the distinction between "Exclusive" and "Key-Shared" is where the real power and complexity lie.
Let’s see it in action. Imagine we have a Pulsar topic named persistent://public/default/my-topic. We’re going to create two different subscription types on this topic and then publish some messages.
First, the "Exclusive" subscription. This is the default and simplest mode.
# Create an exclusive subscription named "my-exclusive-sub"
pulsar-admin subscriptions create persistent://public/default/my-topic --subscription-name my-exclusive-sub --type Exclusive
Now, let’s publish some messages.
# Publish a few messages
pulsar-client produce persistent://public/default/my-topic \
--message "hello exclusive 1" \
--message "hello exclusive 2" \
--message "hello exclusive 3"
If we then try to consume from my-exclusive-sub, we’ll get all three messages.
# Consume from the exclusive subscription
pulsar-client consume persistent://public/default/my-topic \
--subscription-name my-exclusive-sub \
--num-messages 3 \
--no-ack
Output will look something like:
hello exclusive 1
hello exclusive 2
hello exclusive 3
Now, for the "Key-Shared" subscription. This is where things get interesting. We’ll create a new subscription, my-key-shared-sub, of type KeyShared.
# Create a key-shared subscription named "my-key-shared-sub"
pulsar-admin subscriptions create persistent://public/default/my-topic --subscription-name my-key-shared-sub --type KeyShared
When publishing messages with keys, Pulsar’s Key-Shared subscription comes into play. Let’s publish messages with different keys.
# Publish messages with keys
pulsar-client produce persistent://public/default/my-topic \
--key keyA \
--message "message for keyA 1"
pulsar-client produce persistent://public/default/my-topic \
--key keyB \
--message "message for keyB 1"
pulsar-client produce persistent://public/default/my-topic \
--key keyA \
--message "message for keyA 2"
pulsar-client produce persistent://public/default/my-topic \
--key keyC \
--message "message for keyC 1"
pulsar-client produce persistent://public/default/my-topic \
--key keyB \
--message "message for keyB 2"
The core problem Pulsar solves with Key-Shared subscriptions is efficient parallel processing of messages that share the same key. In an "Exclusive" subscription, if you had multiple consumers, only one would ever get a message, and it would get all messages for that topic. This creates a bottleneck. Key-Shared subscriptions allow multiple consumers to share a single subscription, with each consumer receiving messages for a subset of the keys.
Internally, when a consumer connects to a Key-Shared subscription, it establishes a "connection" to a specific partition of the topic. Pulsar then distributes messages belonging to that partition’s keys to that consumer. If multiple consumers are attached to the same Key-Shared subscription, Pulsar will distribute the keys (and their associated messages) across these consumers. For example, if you have two consumers, Consumer 1 might get all messages for keyA and keyC, while Consumer 2 gets all messages for keyB. The exact distribution is managed by Pulsar’s broker. Crucially, if a consumer disconnects, Pulsar can rebalance the keys to other active consumers.
This pattern is incredibly powerful for use cases where messages can be logically grouped by a key, such as processing events for specific users, devices, or transactions. By using Key-Shared subscriptions, you can scale out your consumers to process these key-grouped messages in parallel without introducing separate subscriptions for each key, which would be unmanageable.
To see this in action, you’d typically run multiple pulsar-client consume commands simultaneously, each attached to the my-key-shared-sub. You’d observe that messages with the same key are consistently delivered to the same consumer instance, while messages with different keys are distributed across different consumer instances.
The magic of Key-Shared subscriptions lies in how they manage the mapping between keys and consumers. When a consumer connects, it effectively claims a set of keys for which it will be responsible. This claim is managed by the broker, which keeps track of which consumer is handling which keys within a given partition. If a consumer disconnects, its keys are released, and Pulsar attempts to reassign them to other active consumers for that subscription. This dynamic rebalancing ensures that processing continues even in the face of consumer failures or scaling events.
The next logical step in understanding Pulsar’s consumption patterns is exploring the Shared subscription type, which offers a different approach to distributing messages across multiple consumers.