The Pulsar broker is failing to find metadata for a persistent topic, meaning it can’t locate or access the topic’s ledger in the BookKeeper cluster.
Cause 1: Topic Not Yet Created
Diagnosis:
Check if the topic actually exists using pulsadmin.
/opt/pulsar/bin/pulsadmin topics list my-tenant/my-namespace
Fix: If the topic is missing, create it.
/opt/pulsar/bin/pulsadmin topics create my-tenant/my-namespace/my-topic
Why it works: This explicitly tells Pulsar to create the necessary metadata and BookKeeper ledgers for the topic.
Cause 2: Incorrect Topic Naming Convention
Diagnosis:
Verify that the topic name used by your client (producer/consumer) exactly matches the topic name as registered with pulsadmin or as expected by Pulsar. This includes tenant, namespace, and topic name, along with any partition suffix if applicable (e.g., my-topic-partition-0). Case sensitivity matters.
/opt/pulsar/bin/pulsadmin topics list my-tenant/my-namespace
Compare this output character for character with what your client is configured to use.
Fix: Update your producer or consumer configuration to use the exact, correct topic name.
Why it works: Pulsar uses exact string matching for topic names. A single typo or case difference means the broker can’t find the topic’s metadata.
Cause 3: ZooKeeper Session Expiration or Network Partition
Diagnosis:
Check the Pulsar broker logs for ZooKeeper session expired or connection refused errors related to ZooKeeper. Also, check the BookKeeper bookie logs for any Lost connection to ZooKeeper messages.
On the broker, check its ZooKeeper connection status:
echo "stat" | nc <zookeeper-host> 2181 | grep Mode
Look for standalone, leader, or follower. If it’s not connected, the broker won’t know about topics.
Fix: If ZooKeeper is down or unreachable, address the ZooKeeper cluster’s health. If it’s a transient network issue, ensure network connectivity between brokers, bookies, and ZooKeeper. Restarting the affected Pulsar broker can sometimes resolve transient ZooKeeper connection issues if the ZooKeeper cluster itself is healthy.
Why it works: Pulsar brokers rely on ZooKeeper to discover and manage topics, their metadata, and their underlying BookKeeper ledgers. A broken ZooKeeper connection prevents the broker from accessing this vital information.
Cause 4: BookKeeper Ensemble Size Mismatch or Bookie Failure
Diagnosis:
Check the Pulsar broker logs for errors indicating it cannot find or access the BookKeeper ensemble for the topic. This might manifest as No available bookies or specific ledger access errors.
Examine BookKeeper logs (bookkeeper.log) on the bookies for errors related to ledger access or network issues.
You can inspect the configured ensemble size for a topic (if it exists) by looking at the topic’s metadata in ZooKeeper:
/opt/pulsar/bin/zkCli.sh ls /managed-ledgers/<ledger-id>
(You’ll need to find the ledger ID from broker logs or by inspecting topic metadata in ZK under /admin/persistent/my-tenant/my-namespace/my-topic/partitions/<partition-id>/ledger)
Fix:
Ensure that at least ensembleSize bookies are running and healthy. If a bookie failed, bring it back online or replace it. If the topic was created with a specific ensemble size and some bookies are down, Pulsar might struggle to find a quorum. For critical topics, ensure your BookKeeper cluster has sufficient redundancy (writeQuorum, ackQuorum). If a bookie is permanently lost, you may need to manually force ledger recovery (advanced, use with extreme caution).
Why it works: Persistent topics are stored as ledgers in BookKeeper. If the bookies holding the ledger data or the bookies designated for the topic’s ensemble are unavailable, the broker cannot read or write to the topic.
Cause 5: Topic is being force-deleted or has expired
Diagnosis: Check the Pulsar broker logs for messages indicating that the topic is being force-deleted or has expired due to inactivity and a configured TTL.
You can check the topic’s last access time and configuration:
/opt/pulsar/bin/pulsadmin topics stats my-tenant/my-namespace/my-topic
Look for lastActiveAccessTime and any configured retentionPolicies.
Fix:
If the topic was force-deleted, it needs to be recreated. If it expired due to TTL and you need it back, recreate it. If you want to prevent this, adjust the retentionPolicies for the namespace or topic.
Why it works: Pulsar can automatically delete topics that haven’t been accessed for a configured period to save resources. If a topic is deleted, it ceases to exist and will result in a "Not Found" error until recreated.
Cause 6: Broker Configuration Issues (e.g., wrong ZooKeeper/BookKeeper address)
Diagnosis:
Inspect the conf/broker.conf file on the affected Pulsar broker. Verify that zookeeperServers and bookkeeperClientExceedsConnectionTimeout (and related BookKeeper connection settings) are correctly pointing to your ZooKeeper ensemble and BookKeeper cluster.
Fix:
Correct the zookeeperServers and BookKeeper client configuration in conf/broker.conf to reflect the actual addresses of your ZooKeeper and BookKeeper instances. Restart the Pulsar broker after making changes.
Why it works: The broker needs to know how to connect to ZooKeeper to find topic metadata and to BookKeeper to access topic data. Incorrect connection strings mean it can’t find anything.
The next error you’ll likely hit is a Topic not found error for the internal Pulsar admin topics if the broker itself is unable to connect to ZooKeeper.