The Pulsar broker failed to register with ZooKeeper because its configured brokerServiceUrl was not discoverable by other components.
Common Causes and Fixes
-
Incorrect
brokerServiceUrlinbroker.conf:- Diagnosis: Check the
brokerServiceUrlsetting in your Pulsar broker’s configuration file (conf/broker.conf).grep brokerServiceUrl conf/broker.conf - Fix: Ensure
brokerServiceUrlis set to the IP address and port that other brokers and clients can reach. If your broker is running on192.168.1.100and listening on port1665for internal communication, it should be:
This allows other Pulsar components to know how to connect to this specific broker instance.brokerServiceUrl=pulsar://192.168.1.100:1665 - Why it works: This explicitly tells the Pulsar cluster how to address this broker for inter-broker communication and client connections.
- Diagnosis: Check the
-
advertisedAddressoradvertisedListenersMisconfiguration:- Diagnosis: If
brokerServiceUrlis not explicitly set, Pulsar attempts to derive it fromadvertisedAddressoradvertisedListeners. Check these settings inconf/broker.conf.grep advertisedAddress conf/broker.conf grep advertisedListeners conf/broker.conf - Fix: If using
advertisedAddress, set it to the broker’s publicly accessible IP or hostname:
If usingadvertisedAddress=192.168.1.100advertisedListeners(which is more flexible for different protocols and ports), ensure it includes the correctpulsar://orpulsar+ssl://entry for the broker’s service port. For example:
Then, ensureadvertisedListeners=pulsar://192.168.1.100:1665brokerServiceUrlis either unset or matches this configuration. - Why it works: These settings provide a mechanism for the broker to advertise its network identity to the rest of the cluster, especially in dynamic or multi-network environments.
- Diagnosis: If
-
Firewall Blocking Broker Port:
- Diagnosis: Verify that the
brokerServiceUrlport (default1665) is open on the broker’s host and accessible from other nodes in the cluster. Usetelnetorncfrom another broker or a client machine.telnet 192.168.1.100 1665 # or nc -vz 192.168.1.100 1665 - Fix: Configure your firewall (e.g.,
firewalld,iptables, or cloud security groups) to allow inbound TCP traffic on port1665(or your custombrokerServiceUrlport) from the IP addresses of your other Pulsar nodes and clients.firewalldexample:sudo firewall-cmd --zone=public --add-port=1665/tcp --permanent sudo firewall-cmd --reloadiptablesexample:sudo iptables -A INPUT -p tcp --dport 1665 -j ACCEPT sudo service iptables save
- Why it works: Network connectivity is essential. If other brokers or ZooKeeper cannot reach the
brokerServiceUrl, they cannot establish or maintain connections, leading to registration failures.
- Diagnosis: Verify that the
-
ZooKeeper Not Reachable or Incorrectly Configured:
- Diagnosis: While not directly
brokerServiceUrl, ZooKeeper is where brokers register. If ZooKeeper is inaccessible, the broker can’t register. CheckzookeeperServersinconf/broker.conf.
Then, test connectivity from the broker node to the ZooKeeper ensemble.grep zookeeperServers conf/broker.conftelnet zk1.example.com 2181 - Fix: Ensure
zookeeperServerspoints to the correct ZooKeeper ensemble addresses and ports. If there’s a network issue, open the ZooKeeper port (default2181) in the firewall.zookeeperServers=zk1.example.com:2181,zk2.example.com:2181,zk3.example.com:2181 - Why it works: The broker needs to communicate with ZooKeeper to register itself and discover other cluster members. If this communication fails, the broker cannot join the cluster.
- Diagnosis: While not directly
-
Hostname Resolution Issues:
- Diagnosis: If
brokerServiceUrloradvertisedAddressuses hostnames, ensure these hostnames resolve correctly to the broker’s IP address on all nodes in the cluster (brokers, clients, ZooKeeper). Usepingornslookup.ping broker.example.com nslookup broker.example.com - Fix: Correct your DNS records or update the
/etc/hostsfile on all relevant nodes to map the hostname to the correct IP address. For example, in/etc/hosts:192.168.1.100 broker.example.com - Why it works: Incorrect DNS resolution means other components cannot translate the advertised hostname into a routable IP address, preventing them from connecting to the broker.
- Diagnosis: If
-
Internal Network Routing Problems:
- Diagnosis: Even if ports are open and DNS is correct, underlying network routes might prevent communication between the broker’s advertised IP and other cluster members. Use
traceroutefrom another node to the broker’sbrokerServiceUrlIP.traceroute 192.168.1.100 - Fix: Work with your network administrator to correct any routing misconfigurations between the nodes. This might involve updating gateway configurations or network policies.
- Why it works: Packets need a valid path to travel. If network devices between nodes don’t know how to reach the destination IP, communication will fail silently or with network-level errors.
- Diagnosis: Even if ports are open and DNS is correct, underlying network routes might prevent communication between the broker’s advertised IP and other cluster members. Use
After fixing the brokerServiceUrl and ensuring network connectivity, you might encounter the next error: "ZooKeeper session expired" if the broker was unable to establish a stable connection to ZooKeeper due to the previous configuration issues.