Redis Streams are a powerful data structure for handling ordered sequences of events, and consumer groups take this a step further by enabling multiple consumers to collaboratively process a stream without duplicate messages.
Let’s see a consumer group in action. Imagine we have a stream named sensor_data where we’re publishing readings from various devices.
# Publish a few messages
redis-cli XADD sensor_data * sensor_id 1 temperature 25.5
redis-cli XADD sensor_data * sensor_id 2 humidity 60
redis-cli XADD sensor_data * sensor_id 1 temperature 26.1
Now, let’s create a consumer group called processing_group for this stream.
redis-cli XGROUP CREATE sensor_data processing_group 0
# Output: 1) OK
The 0 here signifies that the group will start processing from the very beginning of the stream if no messages have been consumed by this group yet. If messages were already present and consumed, you’d typically use $ to start from the latest message.
Now, a consumer, let’s call it consumer_1, can join this group and start fetching messages.
# Consumer 1 fetches messages, requesting up to 5 messages
redis-cli XREADGROUP GROUP processing_group consumer_1 COUNT 5 STREAMS sensor_data >
The > is crucial here. It tells Redis to only deliver messages that have not yet been delivered to any consumer within the processing_group. The output might look like this:
1) 1) "sensor_data"
2) 1) 1) "1678886400000-0"
2) 1) "sensor_id"
2) "1"
3) "temperature"
4) "25.5"
2) 1) "1678886410000-0"
2) 1) "sensor_id"
2) "2"
3) "humidity"
4) "60"
3) 1) "1678886420000-0"
2) 1) "sensor_id"
2) "1"
3) "temperature"
4) "26.1"
The consumer receives the message IDs (1678886400000-0, etc.) and their associated key-value pairs. Once consumer_1 has successfully processed these messages, it needs to acknowledge them. This is done using XACK.
# Acknowledge the processed messages
redis-cli XACK sensor_data processing_group 1678886400000-0 1678886410000-0 1678886420000-0
# Output: (integer) 3
This XACK command tells Redis that these specific messages are done. If another consumer in the same group tries to fetch messages with >, it won’t get these acknowledged ones.
The magic of consumer groups is that if consumer_1 crashes before acknowledging a message, that message doesn’t disappear. It enters a "pending" state. Redis keeps track of which messages have been delivered to which consumer within a group, and for how long they’ve been pending. This is managed internally by Redis and can be inspected using XPENDING.
# Check pending messages for the group
redis-cli XPENDING sensor_data processing_group
# Output:
# 1) "-1" (Lower bound of message IDs in the pending list)
# 2) "inf" (Upper bound of message IDs in the pending list)
# 3) "0" (Total number of messages in the pending list)
# 4) (empty list or dict with consumer info if there are pending messages)
If a consumer crashes or is unresponsive for too long, another consumer can "claim" those pending messages using XCLAIM. This allows for fault tolerance and ensures that no message is lost.
# Example of claiming a pending message (if consumer_1 crashed)
redis-cli XCLAIM sensor_data processing_group another_consumer 300000 1678886400000-0
# The 300000 is the minimum idle time in milliseconds after which a message can be claimed.
The core problem Redis Streams consumer groups solve is providing a robust, distributed, and fault-tolerant way to process message queues. They prevent duplicate processing by tracking delivery status per consumer and per group. The internal state management of pending messages and the ability to claim them are what make them resilient to consumer failures.
What most people don’t realize is that the > in XREADGROUP doesn’t just mean "new messages"; it means "messages that have never been delivered to any consumer in this group." Once a message is delivered to any consumer in the group, it’s no longer eligible for > for any consumer in that group, even if the original consumer crashes before acknowledging it. That message then enters the pending state, and subsequent reads by other consumers in the group will not see it until it’s explicitly acknowledged or claimed.
The next major concept to grasp is how to manage the stream’s history effectively, specifically when to trim old messages to prevent the stream from growing indefinitely.