RabbitMQ Shovel and Federation are both tools for moving messages between brokers, but they solve fundamentally different problems, and thinking of them as interchangeable is a common pitfall.
Let’s see Shovel in action. Imagine you have a primary RabbitMQ broker (broker-a) and a secondary, isolated broker (broker-b) that needs to receive specific messages from broker-a for offline processing. You don’t want to replicate the entire message flow, just a subset.
Here’s a basic Shovel configuration in a rabbitmq.conf file, or applied via the management UI:
# Example snippet for rabbitmq.conf (or applied via API/UI)
# This is conceptual; actual config is usually JSON or YAML via API/CLI.
# Shovel definition (conceptual)
shovel_plugins.shovels = my_shovel_to_b
my_shovel_to_b.src-uri = amqp://userA:passA@broker-a:5672/%2F
my_shovel_to_b.src-protocol = amqp0-9-1
my_shovel_to_b.src-exchange = direct_exchange_a
my_shovel_to_b.src-exchange-key = important_message
my_shovel_to_b.src-queue = # The source is an exchange, so we don't specify a queue here for direct routing.
my_shovel_to_b.dest-uri = amqp://userB:passB@broker-b:5672/%2F
my_shovel_to_b.dest-protocol = amqp0-9-1
my_shovel_to_b.dest-exchange = amq.direct
my_shovel_to_b.dest-exchange-key = processed_message
my_shovel_to_b.dest-queue = # Messages are routed directly to an exchange on the destination.
my_shovel_to_b.reconnect-delay = 5
my_shovel_to_b.ack-mode = on-confirm
In this setup, my_shovel_to_b is configured to:
- Connect to
broker-aas the source. - Listen to messages published to
direct_exchange_awith the routing keyimportant_message. - Connect to
broker-bas the destination. - Publish these messages to
amq.directonbroker-bwith the routing keyprocessed_message.
The ack-mode: on-confirm ensures that the message is only removed from broker-a’s queue after it has been successfully acknowledged by broker-b. This provides a strong guarantee for the message’s journey.
Now, let’s contrast this with Federation. Federation is designed for replicating an entire exchange or queue, typically for load balancing, high availability, or creating a geographically distributed cluster. It’s about mirroring a part of one broker’s topology onto another.
Consider a scenario where you have a central broker-prime and several satellite brokers (broker-satellite-1, broker-satellite-2) that need to mirror a specific upstream exchange.
A Federation upstream configuration on broker-satellite-1 might look like this:
# Example snippet for rabbitmq.conf (or applied via API/UI)
# This is conceptual; actual config is usually JSON or YAML via API/CLI.
# Federation upstream definition (conceptual)
federation_upstream_plugins.upstreams = prime_exchange_upstream
prime_exchange_upstream.uri = amqp://prime_user:prime_pass@broker-prime:5672/%2F
prime_exchange_upstream.expires = 3600000 # 1 hour
prime_exchange_upstream.prefetch-count = 1000
prime_exchange_upstream.trust-user-id = false
# Federation policy definition (conceptual)
federation_policy_plugins.policies = replicate_my_exchange
replicate_my_exchange.pattern = ^my_application_exchange$
replicate_my_exchange.definition.federation-upstream = prime_exchange_upstream
replicate_my_exchange.definition.apply-to = exchanges
With this configuration:
prime_exchange_upstreamdefines the connection details tobroker-prime.- The
replicate_my_exchangepolicy is applied tobroker-satellite-1. - The
pattern^my_application_exchange$targets only exchanges named exactlymy_application_exchange. apply-to: exchangesmeans this policy will replicate the specified exchanges.- When messages arrive at
my_application_exchangeonbroker-prime, Federation onbroker-satellite-1will create a mirrored exchange and publish incoming messages to it.
The key difference is the scope and intent. Shovel is about moving specific messages from point A to point B, often with complex routing rules and independent configurations. Federation is about mirroring topology (exchanges/queues) from an upstream broker to a downstream broker, where the downstream broker dynamically adapts its topology based on the upstream.
Shovel is ideal for scenarios like:
- Data Archiving: Moving specific message types to a separate broker for long-term storage.
- Data Transformation: Routing messages to a broker that performs a different kind of processing.
- Disaster Recovery: Replicating critical queues to a standby broker.
- Bridging Heterogeneous Systems: Connecting RabbitMQ to other message brokers or systems that can consume AMQP messages.
Federation excels at:
- Load Balancing: Distributing message processing across multiple consumers connected to different brokers.
- High Availability: Ensuring that if one broker goes down, another can seamlessly take over the load for a replicated exchange or queue.
- Geographic Distribution: Creating a distributed messaging system where data flows between regional brokers.
- Centralized Management: Managing a large number of satellite brokers from a central "prime" broker.
A crucial aspect of Federation is its dynamic nature. When you declare an exchange on the upstream broker, Federation on the downstream broker automatically creates a corresponding mirrored exchange. If you declare a queue bound to that replicated exchange on the upstream, Federation can be configured to replicate that queue as well (using apply-to: queues in the policy). This mirroring is what allows for seamless failover and load distribution.
The most surprising thing about Federation is how it handles queue replication. While it can mirror exchanges directly, mirroring queues requires an explicit policy with apply-to: queues. When a queue is mirrored, Federation creates a new queue on the downstream broker and establishes a binding between the mirrored exchange and this new queue, mimicking the binding that exists on the upstream. This means consumers on the downstream broker can connect to the local mirrored queue and receive messages that originated from the upstream, without needing to know about the upstream broker at all.
If you configure Shovel to pull from an exchange and push to another exchange, but forget to create a queue on the destination exchange that matches the dest-exchange-key, messages will be published but immediately dropped by the destination broker because there’s no binding to consume them.