Shovels in RabbitMQ aren’t just about moving messages; they’re about building resilient, distributed messaging architectures by replicating data between brokers.
Let’s see it in action. Imagine you have a primary RabbitMQ cluster (broker-a) and you want a warm standby or a read replica on another cluster (broker-b). You can set up a shovel to continuously copy messages from a queue on broker-a to a queue on broker-b.
Here’s a simplified shovel.conf for the source broker (broker-a):
# /etc/rabbitmq/shovel.conf
# Source broker connection
source-uri = amqp://user:password@broker-a.example.com:5672/vhost
# Destination broker connection
dest-uri = amqp://user:password@broker-b.example.com:5672/vhost
# The source queue on broker-a
source-queue = my-primary-queue
# The destination queue on broker-b
dest-queue = my-replicated-queue
# How often to check for new messages (in milliseconds)
prefetch-count = 100
reconnect-delay = 5000
Once this configuration is in place and the RabbitMQ Shovel plugin is enabled, you’d start the shovel process. It will then continuously poll my-primary-queue on broker-a, consume messages, and publish them to my-replicated-queue on broker-b. The key is that the shovel consumes from the source and publishes to the destination, effectively moving a copy.
The problem shovels solve is multi-faceted. Primarily, it’s about disaster recovery. If broker-a goes down, broker-b has a recent copy of your messages, minimizing data loss. Secondly, it enables load balancing or read scaling. Consumers can read from broker-b without impacting the performance of broker-a, which is handling writes. It also facilitates data migration between different RabbitMQ environments or even different message brokers if you use more advanced shovel configurations.
Internally, a shovel is a consumer and a producer rolled into one. It connects to the source broker, declares the source queue (or binds to an exchange), and starts consuming messages. For each message it receives, it then connects to the destination broker and publishes that message to the destination queue (or exchange). The shovel manages acknowledgements carefully: a message is only acknowledged on the source after it has been successfully published to the destination and acknowledged by the destination broker. This ensures at-least-once delivery semantics from source to destination.
You control the shovel’s behavior through its configuration file. The source-uri and dest-uri define the connections. source-queue and dest-queue specify the endpoints. prefetch-count dictates how many messages the shovel tries to fetch at once, balancing throughput with memory usage on the shovel process. reconnect-delay is the pause between attempts to re-establish a broken connection. You can also configure shovels to work with exchanges and routing keys, making them far more flexible than simple queue-to-queue replication. For instance, you could shovel messages from an exchange on broker-a to a queue on broker-b using a specific routing key.
The most surprising thing about shovels is how they handle durable messages. When a shovel consumes a durable message from a durable queue on the source, it doesn’t just copy the payload. It reconstructs the message on the destination, including its delivery mode (persistent or transient), content type, headers, and routing key if applicable. This means the replicated message on broker-b behaves identically to the original on broker-a from a consumer’s perspective, preserving crucial metadata that might otherwise be lost in a simpler copy operation.
The next step after setting up a basic shovel is exploring its advanced features like dead-lettering and dynamic shovels.