When a RabbitMQ cluster’s queues aren’t distributing their leaders evenly, you get performance bottlenecks and potential single points of failure.

Let’s see this in action. Imagine a three-node cluster: rabbit1, rabbit2, rabbit3. We have a queue named my_distributed_queue.

# On rabbit1
rabbitmqctl cluster_status

You might see something like this, where rabbit1 is hosting the leader for my_distributed_queue and its mirrors:

{
    "cluster_name": "my-rabbit-cluster",
    "nodes": {
        "rabbit1@example.com": {
            "type": "disc",
            "running_applications": [
                "rabbit",
                "mnesia"
            ],
            "enabled_plugins": [
                "rabbitmq_management",
                "rabbitmq_peer_discovery_k8s",
                "rabbitmq_auth_backend_internal",
                "rabbitmq_amqp1_0",
                "rabbitmq_management_agent",
                "rabbitmq_federation_management",
                "rabbitmq_federation",
                "rabbitmq_peer_discovery_classic_config",
                "rabbitmq_shovel",
                "rabbitmq_stomp",
                "rabbitmq_web_stomp",
                "rabbitmq_amqp0_9_1"
            ]
        },
        "rabbit2@example.com": {
            "type": "disc",
            "running_applications": [
                "rabbit",
                "mnesia"
            ],
            "enabled_plugins": [...]
        },
        "rabbit3@example.com": {
            "type": "disc",
            "running_applications": [
                "rabbit",
                "mnesia"
            ],
            "enabled_plugins": [...]
        }
    },
    "alarms": {},
    "listeners": [
        {
            "port": 5672,
            "protocol": "amqp",
            "ip_address": "0.0.0.0"
        },
        ...
    ],
    "queue_health_indicators": {},
    "network_partitions": [],
    "global_default_user": "guest",
    "global_default_permissions": [
        {
            "vhost": "/",
            "user": "guest",
            "conf": ".*",
            "write": ".*",
            "read": ".*"
        }
    ],
    "auth_backends": [
        "internal"
    ],
    "auth_mechanisms": [
        "PLAIN",
        "AMQPLAIN",
        "EXTERNAL"
    ],
    "log_levels": {
        "/": "info"
    },
    "vm_memory_high_watermark": 0.4,
    "disk_free_limit": 50000000,
    "max_file_descriptors": 1048576,
    "max_tcp_per_node": 2048,
    "disk_used_as_percentage": {},
    "mem_used_as_percentage": {},
    "fd_used_as_percentage": {},
    "auth_required": "elsewhere",
    "queue_master_locator": "balanced",
    "topic_partitions": {}
}

Notice the queue_master_locator is set to balanced. This tells RabbitMQ to try and spread the queue leaders across the cluster.

The problem arises when, despite this setting, leaders get concentrated. This usually happens after node restarts, network issues, or manual intervention that doesn’t respect the intended distribution.

Here’s how the balanced locator works under the hood: When a queue is declared, RabbitMQ’s internal logic considers the current state of the cluster. It looks at which nodes are available and how many queue leaders they are currently hosting. It then attempts to assign the leader role for the new queue to a node that has fewer leaders, aiming for an even distribution. If a node goes down and a queue leader was on that node, its mirrors will elect a new leader. If the cluster is healthy and balanced is configured, this new leader election should ideally pick a node that isn’t already overloaded with leaders.

The queue_master_locator setting is a global configuration for the entire RabbitMQ cluster. You set it once, and it influences all subsequent queue declarations. If you have existing queues and change this setting, it does not automatically rebalance them.

To achieve a truly balanced distribution, you need to understand that RabbitMQ doesn’t have a "rebalance now" button for existing queues. The balanced locator is a declarative setting. It tells the system how to behave going forward. When a queue leader fails, the remaining mirrors will elect a new leader. If balanced is set, the election process will favor nodes with fewer current leaders.

The one thing most people don’t realize is that balanced queue master locator doesn’t actively migrate existing queue leaders. It only influences the election of new leaders when an old one fails, or when new queues are declared. If your cluster has been running for a while and you’ve had node failures or manual interventions, you might have a non-ideal distribution of leaders even with balanced set. The system wants to be balanced, but it needs a trigger event to re-evaluate.

The next logical step is to understand how to manually trigger a re-evaluation of queue leader placement when balanced isn’t doing enough on its own.

Want structured learning?

Take the full Rabbitmq course →