Authorization is failing because the Pulsar broker is unable to verify the identity or permissions of the client attempting to access a topic. This often manifests as an AuthorizationException.
Here are the common culprits and how to fix them:
1. Incorrect Broker Authentication Plugin Configuration
The broker might be configured to use an authentication plugin that doesn’t match how clients are authenticating, or the plugin itself is misconfigured.
Diagnosis:
Check the broker.conf file (usually in /etc/pulsar/ or similar). Look for the authenticationProviders and authenticationService settings. For example, if you’re using TLS authentication, you’d expect something like:
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
authenticationService=org.apache.pulsar.broker.authentication.AuthenticationServiceTls
If you’re using basic auth, it might be:
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderBasicToken
authenticationService=org.apache.pulsar.broker.authentication.AuthenticationServiceBasicToken
Also, check the superUsers configuration to ensure your administrative user is correctly listed.
Fix:
Ensure the authenticationProviders and authenticationService in broker.conf correctly reflect the authentication method used by your clients. If using TLS, ensure the tlsCertificates and tlsKey paths are correct and accessible by the Pulsar user. If using token-based authentication, verify the token secret key is correctly set in broker.conf (tokenSecretKey) and that clients are using a token signed with this same key.
Why it works: The broker needs to know how to interpret the credentials presented by the client. If it’s expecting a TLS certificate but gets a username/password, or if the token secret doesn’t match, authentication will fail, leading to authorization errors.
2. Mismatched TLS Certificates or Truststores
If you’re using TLS authentication, the broker might not trust the client’s certificate, or the client might not trust the broker’s certificate.
Diagnosis:
On the broker side, check broker.conf for tlsTrustCertsFilePath. This should point to the truststore containing the CA that signed the client certificates. On the client side, check its configuration for tlsTrustCertsFilePath (or equivalent) which should point to the truststore containing the CA that signed the broker certificate.
Verify the validity period and issuer of the certificates being presented.
Fix:
Ensure that the tlsTrustCertsFilePath on the broker contains the CA certificate that signed the client’s certificate. Similarly, the client’s truststore must contain the CA certificate that signed the broker’s certificate. If you’re using self-signed certificates, you need to manually add the CA certificate (or the certificate itself if it’s self-signed and used for both client and server) to the respective truststores. For example, using keytool:
# Add broker's CA cert to client's truststore
keytool -importcert -alias pulsar-broker-ca -file /path/to/broker_ca.pem -keystore /path/to/client.truststore.jks -storepass <password>
# Add client's CA cert to broker's truststore
keytool -importcert -alias pulsar-client-ca -file /path/to/client_ca.pem -keystore /path/to/broker.truststore.jks -storepass <password>
Restart the broker and clients after updating truststores.
Why it works: TLS relies on a chain of trust. If the broker doesn’t recognize the CA that vouches for the client’s identity, it will reject the connection.
3. Incorrectly Configured Authorization Provider (e.g., ZookeeperAclProvider)
The ZookeeperAclProvider (or other ACL providers) is responsible for checking if an authenticated user has permission to perform an action on a specific resource. If this provider is misconfigured, it can deny legitimate requests.
Diagnosis:
In broker.conf, check the authorizationProvider setting. If it’s org.apache.pulsar.broker.authorization.ZookeeperAclProvider, then Pulsar is using ZooKeeper for authorization rules.
Check ZooKeeper for the ACLs. You can use zkCli.sh (from ZooKeeper’s bin directory) to inspect ACLs. For example, to list ACLs for a topic my-topic in namespace my-tenant/my-namespace:
# Connect to ZooKeeper
./zkCli.sh -server <zk_host>:2181
# List ACLs for the topic
getAcl /admin/persistent/my-tenant/my-namespace/my-topic
Fix:
Ensure that the authenticated user has the necessary permissions (e.g., subscriptions for consuming, produce for producing) on the topic or namespace. You can grant permissions using the Pulsar Admin CLI:
# Grant produce permission to user 'my-user' on namespace 'my-tenant/my-namespace'
pulsar-admin namespaces grant-permission my-tenant/my-namespace \
--actions produce \
--principal my-user
# Grant subscription permission to user 'my-user' on topic 'persistent://my-tenant/my-namespace/my-topic'
pulsar-admin topics grant-permission persistent://my-tenant/my-namespace/my-topic \
--actions subscriptions \
--principal my-user
Ensure the broker.conf has the correct ZooKeeper connection string (zookeeperServers).
Why it works: The ZookeeperAclProvider reads authorization rules from ZooKeeper. If the rules are missing, incorrect, or the broker can’t connect to ZooKeeper to read them, it will default to denying access.
4. Client Not Providing Credentials or Providing Invalid Credentials
The client might be configured to authenticate but is not sending any credentials, or the credentials it’s sending are malformed or expired.
Diagnosis:
Examine the client’s configuration (e.g., producer.conf, consumer.conf, or programmatically set properties).
For token authentication, check if auth_plugin=org.apache.pulsar.client.impl.auth.AuthenticationToken and auth_params=token:your_token_here are set.
For TLS authentication, check if tlsAllowInsecureConnections=false and tlsTrustCertsFilePath are correctly configured, and that the client has access to its own certificate and private key.
Check the Pulsar broker logs for specific authentication errors related to the client’s connection attempt.
Fix:
Ensure the client is configured with the correct auth_plugin and auth_params that match the broker’s authenticationProviders.
If using tokens, generate a new, valid token. Ensure the token is not expired and is signed with the same secret key the broker is configured with.
If using TLS, ensure the client is configured with its certificate and private key, and that these are valid and trusted.
Why it works: The broker’s authentication mechanism needs valid credentials to identify the client. Without them, or with invalid ones, it cannot proceed to authorization.
5. Incorrect Tenant/Namespace Configuration or Missing Permissions
The tenant or namespace the client is trying to access might not exist, or the user might not have been granted permissions at the tenant or namespace level.
Diagnosis: Use the Pulsar Admin CLI to check the existence of the tenant and namespace:
pulsar-admin tenants list
pulsar-admin namespaces list my-tenant
Then, check the permissions for the user on that namespace:
pulsar-admin namespaces get-permissions my-tenant/my-namespace
Fix:
Create the tenant and/or namespace if they don’t exist. Grant the necessary permissions to the authenticated user on the namespace. For example, to allow a user my-user to produce and consume from all topics in my-tenant/my-namespace:
pulsar-admin namespaces grant-permission my-tenant/my-namespace \
--actions produce,consume \
--principal my-user
Ensure the principal matches the authenticated identity of the client.
Why it works: Pulsar’s authorization model is hierarchical. Permissions granted at the namespace level apply to all topics within it. If a user lacks permissions at this level, they cannot access any topics unless specific topic-level permissions override it.
6. Broker Not Connected to ZooKeeper or BookKeeper
Authorization information is often stored in ZooKeeper, and topic metadata is in BookKeeper. If the broker cannot connect to these dependencies, it cannot reliably perform authorization checks.
Diagnosis:
Check the broker logs (/var/log/pulsar/pulsar.log or similar) for ZooKeeper or BookKeeper connection errors, timeouts, or connection refused messages.
Verify the zookeeperServers and bookkeeper.client.service-urls settings in broker.conf.
Check network connectivity from the broker to the ZooKeeper and BookKeeper ensemble.
Fix:
Ensure ZooKeeper and BookKeeper are running and accessible from the broker nodes. Correct the zookeeperServers and bookkeeper.client.service-urls in broker.conf if they are incorrect. Resolve any network issues (firewalls, DNS) preventing communication. Restart the Pulsar broker after making changes.
Why it works: The authorization and metadata systems are external dependencies. If the broker cannot reach them, it cannot fetch the necessary information to authorize requests or even know which topics exist.
The next error you’ll likely encounter after fixing authorization is a BrokerServiceException if the underlying topic or partition is not available, or a ServiceConfigurationException if a core service like ZooKeeper is still unreachable.