The Pulsar client is failing to connect to the broker because the underlying network or authentication mechanism is preventing the connection from being established.
Common Causes and Fixes:
-
Broker Not Reachable (Network Connectivity)
- Diagnosis: On the client machine, run
ping <broker_host>. If it fails, or iftraceroute <broker_host>shows unexpected hops or timeouts, network connectivity is the issue. Check firewall rules on both client and broker machines, as well as any intermediate network devices. - Fix: Ensure the Pulsar broker’s advertised listener address (
listenersinbroker.conf) is accessible from the client’s network. If using a specific port, verify it’s open:telnet <broker_host> <broker_port>. - Why it works: This confirms basic IP-level reachability and that the target port is listening and not blocked by a firewall.
- Diagnosis: On the client machine, run
-
Incorrect Broker Listeners Configuration
- Diagnosis: Examine
broker.confon the Pulsar broker. Thelistenersproperty must be set to an address and port that the clients can actually reach. For example, if the broker is running on192.168.1.100and clients are on a different subnet,listeners=PLAINTEXT://192.168.1.100:6650might not work if192.168.1.100is not routable from the client. - Fix: Set
listenersto an IP address that is routable from the client network, or use0.0.0.0if you want to bind to all available interfaces and rely on network routing/firewalls. For example:listeners=PLAINTEXT://<routable_ip_address>:6650. If using TLS, it would belisteners=SSL://<routable_ip_address>:6651. - Why it works: The client needs to resolve and connect to the exact address and port the broker is advertising and listening on.
- Diagnosis: Examine
-
Authentication Failure (e.g., Missing or Invalid Token/Credentials)
- Diagnosis: Check the Pulsar broker logs (
broker.log) for authentication-related errors. Look for messages indicating "Authentication failed," "Invalid credentials," or similar. On the client side, check client logs for connection errors that might include authentication details. - Fix:
- Token Authentication: Ensure the client is configured with a valid token. In the client code, this is typically set via
ClientConfigurationData.setAuthentication(tokenAuthPlugin, tokenAuthParameters). ThetokenAuthParametersshould be a JSON string like{"token": "your-long-jwt-token"}. - Plaintext/Basic Auth: Verify the username and password are correct and that the client configuration matches the broker’s expected authentication method.
- Mutual TLS (mTLS): Ensure the client has the correct keystore and truststore configured, and that the certificates are valid and trusted by the broker.
- Token Authentication: Ensure the client is configured with a valid token. In the client code, this is typically set via
- Why it works: Pulsar brokers can be configured with various authentication mechanisms. The client must present valid credentials according to the broker’s configured
authenticationProviders.
- Diagnosis: Check the Pulsar broker logs (
-
TLS/SSL Handshake Failure (if TLS is enabled)
- Diagnosis: Broker logs will show errors like "SSLHandshakeException" or "Certificate validation failed." Client logs will also indicate TLS handshake issues.
- Fix:
- Broker Side: Ensure the broker’s
tlsTrustCertsFilePathpoints to a valid truststore containing the CA that signed the broker’s certificate, andtlsCertificateFilePathandtlsKeyFilePathpoint to the broker’s valid certificate and private key. - Client Side: Ensure the client’s
tlsTrustCertsFilePathpoints to a truststore containing the CA that signed the broker’s certificate. If the client also needs to present a certificate, itstlsCertificateFilePathandtlsKeyFilePathmust be correctly configured.
- Broker Side: Ensure the broker’s
- Why it works: A successful TLS handshake requires both client and server to trust each other’s certificates, which are validated against configured truststores.
-
ZooKeeper Connectivity Issues
- Diagnosis: While Pulsar clients don’t directly connect to ZooKeeper, brokers do. If brokers cannot connect to ZooKeeper, they may not be able to serve client requests or even start properly, leading to client connection failures. Check broker logs for ZooKeeper connection errors ("ZooKeeper connection lost," "Failed to connect to ZooKeeper").
- Fix: Verify the
zookeeperServerssetting inbroker.confis correct and points to resolvable and reachable ZooKeeper instances. Ensure firewalls between brokers and ZooKeeper are not blocking the ZooKeeper client port (default 2181). Check ZooKeeper server logs for any issues. - Why it works: Brokers rely on ZooKeeper for cluster metadata, leader election, and service discovery. If this communication breaks, the broker’s ability to function is severely impaired.
-
Incorrect Service URL Format
- Diagnosis: Review the service URL being used by the Pulsar client. Common issues include using the wrong protocol (e.g.,
pulsar://when TLS is required, orpulsar+ssl://when it’s not), incorrect hostname/IP, or wrong port. - Fix: Use the correct service URL format.
- For unencrypted connections:
pulsar://<broker_host>:<broker_port>(e.g.,pulsar://pulsar.example.com:6650) - For TLS-encrypted connections:
pulsar+ssl://<broker_host>:<broker_tls_port>(e.g.,pulsar+ssl://pulsar.example.com:6651) - If using a load balancer, use its address.
- For unencrypted connections:
- Why it works: The service URL tells the client which protocol to use and where to find the Pulsar service. Mismatches here prevent the initial connection handshake.
- Diagnosis: Review the service URL being used by the Pulsar client. Common issues include using the wrong protocol (e.g.,
-
Resource Exhaustion on Broker (CPU, Memory, File Descriptors)
- Diagnosis: Monitor the Pulsar broker’s resource utilization. High CPU, memory usage, or exceeding the open file descriptor limit (
ulimit -n) can cause the broker to become unresponsive to new client connections. Checktop,htop,vmstat, anddmesgon the broker machine. - Fix:
- Increase broker resources (CPU, RAM).
- Tune Pulsar configuration parameters related to connection handling, thread pools, or message processing.
- Increase the open file descriptor limit for the user running the Pulsar broker:
ulimit -n 65536(or higher, as needed). Ensure this is persistent by configuring/etc/security/limits.conf.
- Why it works: An overloaded broker cannot accept new connections efficiently, leading to timeouts and connection errors for clients. Increasing file descriptors is crucial as each connection and internal file handle consumes one.
- Diagnosis: Monitor the Pulsar broker’s resource utilization. High CPU, memory usage, or exceeding the open file descriptor limit (
The next error you’ll likely encounter after fixing connectivity and authentication is related to topic lookup failures, often manifesting as Topic not found or Topic not available errors, even if the broker is reachable.