The RabbitMQ trace plugin, often called "firehose," doesn’t just log messages; it fundamentally alters how you reason about message delivery by making every hop visible.
Let’s see it in action. Imagine a simple scenario: a producer sends a message to an exchange, which then routes it to a queue where a consumer picks it up. Without tracing, if a message disappears, you’re left guessing.
Here’s a producer sending a message:
import pika
from pika.exchange_type import ExchangeType
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.exchange_declare(exchange='my_exchange', exchange_type=ExchangeType.direct)
channel.queue_declare(queue='my_queue', durable=True)
channel.queue_bind(exchange='my_exchange', queue='my_queue', routing_key='my_key')
channel.basic_publish(
exchange='my_exchange',
routing_key='my_key',
body='Hello, firehose!'
)
print(" [x] Sent 'Hello, firehose!'")
connection.close()
And a consumer:
import pika
from pika.exchange_type import ExchangeType
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.exchange_declare(exchange='my_exchange', exchange_type=ExchangeType.direct)
channel.queue_declare(queue='my_queue', durable=True)
channel.queue_bind(exchange='my_exchange', queue='my_queue', routing_key='my_key')
def callback(ch, method, properties, body):
print(f" [x] Received {body.decode()}")
channel.basic_consume(queue='my_queue', on_message_callback=callback, auto_ack=False)
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()
Now, let’s enable the trace plugin. First, ensure it’s installed and enabled in your rabbitmq.conf or via the command line. On a typical installation, you might enable it like this:
rabbitmq-plugins enable rabbitmq_tracing
After restarting RabbitMQ, you’ll need to configure which exchanges and queues to trace. This is crucial because tracing everything can overwhelm your system. You configure this via rabbitmqctl:
rabbitmqctl set_tracing producer.* on
rabbitmqctl set_tracing consumer.* on
rabbitmqctl set_tracing exchange.my_exchange on
rabbitmqctl set_tracing queue.my_queue on
The producer.* and consumer.* patterns are powerful. producer.* captures events originating from any producer, and consumer.* captures events related to consumers. exchange.my_exchange and queue.my_queue are specific to the objects.
Once tracing is enabled and configured, run your producer and consumer. The trace plugin will start emitting detailed event logs. You can view these logs in several ways, but the most common is via the RabbitMQ management UI under the "Connections" tab. You’ll see a "Tracing" section. Alternatively, you can subscribe to the trace exchange directly using a dedicated client or even another RabbitMQ client.
The trace exchange is typically named rabbit_trace. To listen to it, you’d declare a temporary, exclusive queue and bind it with a wildcard:
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
# Declare a temporary, exclusive queue
result = channel.queue_declare(exclusive=True)
trace_queue_name = result.method.queue
# Bind to the trace exchange with a wildcard
channel.queue_bind(exchange='rabbit_trace', queue=trace_queue_name, routing_key='#')
print(f" [*] Listening to trace events on queue: {trace_queue_name}")
def trace_callback(ch, method, properties, body):
print(f" [TRACE] {body.decode()}")
channel.basic_consume(queue=trace_queue_name, on_message_callback=trace_callback, auto_ack=True)
channel.start_consuming()
When your producer sends the message, you’ll see events like basic.publish on the rabbit_trace exchange, tagged with routing keys that indicate the source and destination. If you’ve traced the exchange and queue, you’ll see exchange.route events showing how the message arrived at the exchange and then queue.bind or queue.deliver events as it moves towards the queue. When the consumer receives it, you’ll see basic.deliver events on the queue.
This granular visibility is invaluable for debugging. If a message is published but never appears in the queue, the trace logs will show exactly where it was dropped – perhaps a routing key mismatch, a misconfigured binding, or an exchange that doesn’t exist. If a message is in the queue but the consumer doesn’t receive it, the trace will show if it was delivered but not acknowledged, or if the consumer connection was lost before delivery.
The most counterintuitive aspect of the trace plugin is its routing key structure. While you bind to rabbit_trace with #, the actual routing keys on the trace exchange are highly specific, often mimicking the operations themselves, like basic.publish, exchange.route, queue.deliver, channel.close, and connection.close, prefixed by the component that generated the event (e.g., exchange.my_exchange.route). This allows for extremely fine-grained filtering by subscribing to specific routing keys.
Once you’ve debugged your message flow, remember to disable tracing for the components you no longer need to monitor, and eventually disable the plugin entirely if it’s not a permanent requirement, as it does incur performance overhead.
The next hurdle after mastering message flow tracing is understanding how to filter and aggregate these high-volume trace events for more complex diagnostics.