The Pulsar authentication system is failing because client connections are being rejected by the broker, indicating a mismatch between the credentials presented by the client and what the broker expects. This is usually due to configuration errors on either the client or the broker side, or issues with the underlying authentication mechanism.

Common Causes and Fixes for Pulsar Authentication Failed Errors

  1. Incorrect authentication.providers Configuration on Broker:

    • Diagnosis: Check the broker’s conf/broker.conf (or conf/pulsar-shell.conf for pulsar-shell) for the authentication.providers setting. Ensure it’s correctly set to the authentication method you intend to use (e.g., authentication.providers=org.apache.pulsar.broker.authentication.AuthenticationProviderToken).
    • Fix: If the provider is missing or incorrect, add or correct it. For example, to enable token authentication:
      # In conf/broker.conf
      authentication.providers=org.apache.pulsar.broker.authentication.AuthenticationProviderToken
      
    • Why it works: This setting tells the broker which Java class to load and use for validating incoming authentication requests. Without the correct provider, the broker cannot understand or process any authentication credentials.
  2. Mismatched authentication.token.secret-key (for Token Auth):

    • Diagnosis: For token-based authentication, both clients and brokers must use the same secret key for signing and verifying tokens. On the broker side, check conf/broker.conf for authentication.token.secret-key. On the client side, check the client configuration (e.g., conf/client.conf or environment variables) for the equivalent setting.
    • Fix: Ensure the authentication.token.secret-key value is identical on all brokers and all clients using token authentication.
      # In conf/broker.conf and client.conf (or equivalent)
      authentication.token.secret-key=your-super-secret-key
      
    • Why it works: This key is used to cryptographically sign the tokens issued by the authentication provider. If the broker uses a different key to verify the token than the one used to sign it, the signature verification will fail, leading to authentication errors.
  3. Incorrect authentication.jwt.private-key and authentication.jwt.public-key (for JWT Auth):

    • Diagnosis: If using JWT authentication, verify that the authentication.jwt.private-key (used by the token issuer, often an external service or the Pulsar auth provider itself) and authentication.jwt.public-key (used by the broker to verify the token) are correctly configured and that the public key matches the private key. Check conf/broker.conf for authentication.jwt.public-key and the token issuer’s configuration for authentication.jwt.private-key.
    • Fix: Ensure the authentication.jwt.public-key in conf/broker.conf corresponds to the private key used to sign the JWT. The public key should be the content of the public key file (e.g., certs/public.pem).
      # In conf/broker.conf
      authentication.jwt.public-key=file:///path/to/your/public.pem
      
      The private key would be configured where the JWT is generated.
    • Why it works: JWTs are signed with a private key, and verified with a corresponding public key. If the public key configured on the broker doesn’t match the private key used for signing, the JWT’s signature will be invalid.
  4. Client Not Sending Required Credentials:

    • Diagnosis: Clients must be explicitly configured to send authentication credentials. For token authentication, this is typically done via the auth_token setting. For other methods, it might be auth_plugin and related settings. Check client logs for messages indicating missing credentials.
    • Fix: Configure the client to send the correct credentials. For token authentication:
      # In conf/client.conf or programmatically
      auth_plugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
      auth_token=your-client-token
      
      Or via environment variable:
      PULSAR_CLIENT_CONF_FILE=/path/to/client.conf
      export PULSAR_AUTH_TOKEN=your-client-token
      
    • Why it works: The broker is configured to require authentication. If the client doesn’t present any credentials, the broker will reject the connection immediately.
  5. Incorrect authentication.tls.trustCertsFilePath or authentication.tls.certFilePath (for TLS Auth):

    • Diagnosis: If using TLS client authentication, the broker needs to trust the client’s certificate and vice-versa. Check conf/broker.conf for authentication.tls.trustCertsFilePath (path to the CA certificate that signed the client’s certificate) and authentication.tls.certFilePath (path to the broker’s certificate). Also, check client configuration for its certificate and key.
    • Fix: Ensure the authentication.tls.trustCertsFilePath on the broker points to the correct CA certificate bundle that signed the client’s certificate, and that the client is configured with its own certificate and private key.
      # In conf/broker.conf
      authentication.tls.trustCertsFilePath=/path/to/ca.crt
      authentication.tls.certFilePath=/path/to/broker.crt
      authentication.tls.keyFilePath=/path/to/broker.key
      
      Client configuration would similarly specify its certificate and key.
    • Why it works: TLS authentication relies on a chain of trust. The broker must be able to verify that the client’s certificate was issued by a trusted Certificate Authority (CA), and the client must verify the broker’s certificate. Misconfigured trust stores or missing certificate/key pairs break this verification.
  6. Time Skew Between Client and Broker (for JWT/Token Auth with Expiration):

    • Diagnosis: JWTs and some token-based authentication schemes have expiration times. If the system clocks on the machines running the brokers and the clients are significantly out of sync, tokens that are valid on the issuing server might appear expired to the broker, or vice-versa. Check logs for messages indicating expired tokens.
    • Fix: Synchronize the clocks of all Pulsar nodes (brokers, Zookeeper, etc.) and client machines using NTP (Network Time Protocol).
      # On Linux/macOS, ensure ntpd or chronyd is running and configured
      sudo systemctl enable ntp
      sudo systemctl start ntp
      
    • Why it works: JWTs and time-sensitive tokens rely on accurate time for their validity period (nbf - not before, exp - expiration). Significant clock drift can cause valid tokens to be rejected or invalid tokens to be accepted.
  7. Incorrect broker.conf Configuration for superUsers:

    • Diagnosis: If you’re trying to perform administrative operations and get authentication errors, it might be that the user is not in the superUsers list. Check conf/broker.conf for the superUsers setting.
    • Fix: Add the authenticated user’s identity (e.g., username or certificate DN) to the superUsers list.
      # In conf/broker.conf
      superUsers=admin_user,CN=my-admin-cert
      
    • Why it works: Pulsar has a concept of super users who are granted elevated privileges. If authentication succeeds but the user is not recognized as a super user, certain administrative actions will fail.

The next error you’ll likely encounter after fixing authentication is related to authorization, where a user is authenticated but doesn’t have permission to perform a specific action on a topic or namespace.

Want structured learning?

Take the full Pulsar course →