Redis Lists and Streams look similar on the surface, but they solve fundamentally different problems, and picking the wrong one can lead to massive inefficiencies and bugs.
Imagine you’re building a simple task queue. You want to add tasks and then process them one by one. A Redis List seems like the obvious choice.
# Add a task
redis-cli RPUSH my_task_queue "Process user signup for ID 123"
# Add another task
redis-cli RPUSH my_task_queue "Send welcome email to user ID 456"
# Get the oldest task
redis-cli LPOP my_task_queue
# Output: "Process user signup for ID 123"
# Get the next task
redis-cli LPOP my_task_queue
# Output: "Send welcome email to user ID 456"
This works great for simple FIFO (First-In, First-Out) queues. RPUSH adds to the right (end) and LPOP removes from the left (beginning). It’s straightforward, efficient for this use case, and the commands are simple.
But what if you need more? What if you have multiple workers that need to compete for tasks, and you need to know which worker processed which task, and be able to re-queue a task if a worker crashes? This is where Redis Streams shine.
A Redis Stream is an append-only log. Think of it like a Kafka topic, but much simpler and built into Redis. Each entry in a stream has a unique ID, a set of key-value pairs, and critically, consumers can read from it in groups.
Let’s set up a stream and a consumer group:
# Add an entry to the stream
redis-cli XADD my_event_stream * user_id 123 action "signup"
# Add another entry
redis-cli XADD my_event_stream * user_id 456 action "login"
# Create a consumer group for our workers
# '>' means the group will only read new messages appended after group creation
redis-cli XGROUP CREATE my_event_stream my_consumer_group '$' MKSTREAM
# Output: OK
Now, multiple worker instances can read from this stream:
# Worker 1 reads a message from the group
redis-cli XREADGROUP GROUP my_consumer_group worker COUNT 1 STREAMS my_event_stream '>'
# Output:
# 1) 1) "my_event_stream"
# 2) 1) 1) "1678886400000-0" <-- This is the stream entry ID
# 2) 1) "user_id"
# 2) "123"
# 3) "action"
# 4) "signup"
The key here is XREADGROUP. It allows multiple consumers (identified by worker in this example) to form a my_consumer_group. Redis ensures that each message is delivered to only one consumer within that group. If a consumer crashes before acknowledging a message, that message can be "claimed" by another consumer.
This also gives us a robust way to track progress. When a worker finishes processing a message, it acknowledges it:
# Worker 1 acknowledges processing the message
redis-cli XACK my_event_stream my_consumer_group 1678886400000-0
# Output: (integer) 1
If a worker dies, you can use XPENDING to see messages that were delivered but not acknowledged, and XCLAIM to reassign them to another worker. This is impossible with a simple Redis List.
The most surprising true thing about Redis Streams is that their IDs are not just sequential numbers; they are composed of a Unix timestamp in milliseconds and a sequence number. This ensures global ordering across all Redis instances if you’re using Redis Cluster, and provides a natural sorting mechanism. The * in XADD tells Redis to automatically generate this ID for you.
The core difference is that Lists are simple sequence containers, great for basic queues or producer-consumer patterns where you don’t need complex tracking. Streams are append-only logs designed for event sourcing, message queues with competing consumers, and scenarios where you need guaranteed delivery, acknowledgments, and the ability to replay events.
When you use XREADGROUP with the > special ID, you’re telling Redis to only deliver messages that have never been delivered to any consumer in that group. This is crucial for ensuring that each message is processed exactly once by some worker in your group, even if workers come and go.
If you’re building a simple FIFO queue and don’t need advanced features like consumer groups, acknowledgments, or message history, a Redis List with RPUSH/LPOP or LPUSH/RPOP is perfectly adequate and simpler. But if you need robustness, competing consumers, or a reliable event log, Redis Streams are the way to go.
Once you’ve mastered competing consumers with Streams, the next logical step is exploring how to manage stream history and prune old messages using XTRIM.