The Pulsar broker is failing to locate the namespace metadata, preventing clients from accessing topics within it.

Common Causes and Fixes

  1. ZooKeeper Session Expiration/Disconnection:

    • Diagnosis: Check the Pulsar broker logs (/var/log/pulsar/broker.log) for messages like ZooKeeper session expired or ZooKeeper connection lost. Simultaneously, check ZooKeeper logs for connection issues from the broker’s IP. You can also run echo "stat" | nc <zookeeper_host> 2181 and look for the broker’s connection count.
    • Fix: Increase the ZooKeeper session timeout and connection timeout for the Pulsar broker. In broker.conf, set:
      zookeeperSessionTimeoutMs=60000
      zookeeperConnectTimeoutMs=30000
      
      Restart the Pulsar broker.
    • Why it works: Pulsar relies heavily on ZooKeeper for metadata management, including namespace ownership and topic discovery. If the broker’s connection to ZooKeeper is unstable or times out, it loses track of namespace information. Increasing these timeouts provides a more robust connection.
  2. Incorrect ZooKeeper Connection String:

    • Diagnosis: Verify the zookeeperServers setting in broker.conf. Ensure it precisely matches the active ZooKeeper ensemble’s connection string, including all comma-separated hosts and ports (e.g., zk1:2181,zk2:2181,zk3:2181).
    • Fix: Correct the zookeeperServers entry in broker.conf to the accurate ZooKeeper ensemble. Restart the Pulsar broker.
    • Why it works: The broker cannot find any metadata, including namespace information, if it’s not pointing to the correct ZooKeeper cluster.
  3. ZooKeeper Data Corruption or Inconsistency:

    • Diagnosis: If brokers report connectivity but still can’t find namespaces, there might be an issue within ZooKeeper itself. Use ls /admin/namespaces in the ZooKeeper CLI to see if the expected namespace paths (e.g., /admin/namespaces/public/default) exist. If they are missing or appear malformed, ZooKeeper data is suspect.
    • Fix: This is a more drastic measure. If namespace data is truly missing from ZooKeeper, it likely means the cluster was not shut down cleanly or there was a failure during metadata writes. The safest approach is to restore ZooKeeper from a known good backup. If no backup is available, you might need to re-create namespaces via the Pulsar admin API, but this will lose historical data if not done carefully.
    • Why it works: ZooKeeper is the single source of truth for Pulsar’s configuration and state. If its data is corrupted, the broker has no reliable information to operate on.
  4. Broker Not Fully Started/Initialized:

    • Diagnosis: Check the broker.log for any startup errors or warnings. Look for messages indicating that the broker has successfully connected to ZooKeeper and registered itself. The absence of these messages suggests an incomplete startup.
    • Fix: Ensure the broker process is running. If it crashed or failed to start, investigate the underlying cause (e.g., insufficient memory, configuration errors). A simple restart (systemctl restart pulsar-broker or equivalent) might suffice if it was a transient issue.
    • Why it works: A broker that hasn’t fully initialized won’t have loaded or registered its necessary metadata handlers, including those for namespaces.
  5. Network Issues Between Broker and ZooKeeper:

    • Diagnosis: Use ping <zookeeper_host> and nc -vz <zookeeper_host> 2181 from the broker’s server to check basic network connectivity and if the ZooKeeper port is open. Firewalls are a common culprit.
    • Fix: Configure firewall rules (e.g., firewall-cmd --add-port=2181/tcp --permanent on CentOS/RHEL) on both the broker and ZooKeeper nodes to allow traffic on port 2181. Restart the Pulsar broker.
    • Why it works: Even if the zookeeperServers string is correct, the broker cannot communicate with ZooKeeper if network paths are blocked.
  6. Incorrect Pulsar Cluster Configuration:

    • Diagnosis: In broker.conf, check clusterName. This name must exactly match the cluster name defined in the Pulsar configuration that ZooKeeper is using (often found in conf/global/coordinates.conf or similar within the ZooKeeper configuration itself, if Pulsar is managing its own ZooKeeper). If this is a multi-cluster setup, ensure the broker is configured for the correct cluster.
    • Fix: Align the clusterName in broker.conf with the cluster name Pulsar expects in ZooKeeper. Restart the Pulsar broker.
    • Why it works: Pulsar uses the cluster name to partition metadata in ZooKeeper. If the broker is looking for metadata under the wrong cluster identifier, it will fail to find namespaces belonging to its actual cluster.

The next error you’ll likely encounter if all namespace issues are resolved is a Topic not found error, indicating a problem at the topic level rather than the namespace level.

Want structured learning?

Take the full Pulsar course →