Pulsar’s JWT authentication is designed to rotate tokens automatically, but sometimes you need to manually renew them.
Let’s watch a token in action. Imagine a producer sending messages to a topic.
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.authentication(
HttpAuthentication.builder()
.authMethod("token")
.credentials("eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c")
.build()
)
.build();
Producer<String> producer = client.newProducer(Schema.STRING)
.topic("persistent://public/default/my-topic")
.create();
producer.send("Hello Pulsar!");
Here, the credentials field holds the JWT. This token has an expiration time embedded within it. When it expires, the client should automatically request a new one from the authentication provider.
However, this automatic renewal can fail for several reasons.
1. Clock Skew: If the client machine’s clock is significantly out of sync with the broker’s clock, the token might appear expired to the broker even if the client thinks it’s still valid, or vice-versa. This can lead to authentication failures.
- Diagnosis: Check the system time on both the Pulsar broker and the client machines. Use
dateon Linux/macOS orw32tm /query /statuson Windows. - Fix: Synchronize clocks using NTP. On Linux, ensure
ntpdorchronydis running and configured correctly. For example, to synchronize withpool.ntp.org:sudo systemctl start ntp sudo systemctl enable ntp sudo ntpdate pool.ntp.org - Why it works: Ensures consistent time across all nodes, allowing for accurate validation of token
iat(issued at) andexp(expiration) claims.
2. Incorrect Token Signing Key: The broker uses a public key to verify the signature of the JWTs issued by the authentication provider. If this key is incorrect or has been changed without updating the broker configuration, valid tokens will be rejected.
- Diagnosis: Examine the Pulsar broker configuration file (
broker.conforstandalone.conf). Look forjwt.public.key. Verify that this key matches the private key used by your authentication provider. - Fix: Update
broker.confwith the correct public key and restart the Pulsar broker. For example, if using PEM format:# broker.conf jwt.public.key=-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA... -----END PUBLIC KEY----- - Why it works: The broker can now correctly validate the digital signature of the JWT, confirming its authenticity and integrity.
3. Expired or Invalid Authentication Provider Token: The authentication provider itself might be issuing tokens that are already expired or malformed, or its own internal clock might be skewed.
- Diagnosis: Inspect the JWT itself. You can use online JWT decoders (like jwt.io) to view the claims, including
exp. Check theiatandexptimestamps. - Fix: Ensure your authentication provider is configured to issue tokens with sufficient validity periods and that its own system time is accurate. For example, if using a custom provider, ensure its token generation logic is sound and its server clocks are synchronized.
- Why it works: A validly issued token with a future expiration time is essential for establishing a connection.
4. Network Issues Between Client and Authentication Provider: When a client needs a new token, it communicates with the configured authentication provider. Firewalls, network latency, or service unavailability can prevent this communication.
- Diagnosis: On the client machine, try to
curlthe endpoint of your authentication provider. Check firewall rules between the client and the authentication provider’s service. - Fix: Open necessary ports in firewalls or ensure the authentication provider service is running and accessible from the client. For example, if the provider is on
auth.example.com:8080:# On client curl http://auth.example.com:8080/get-token - Why it works: Establishes a reliable communication channel for token issuance.
5. Incorrect token.audience Configuration: The token.audience configuration in Pulsar is used to specify the expected audience for JWTs. If this is misconfigured, valid tokens might be rejected.
- Diagnosis: Compare the
aud(audience) claim in your JWT with thetoken.audiencesetting inbroker.conf. - Fix: Update
broker.confto match theaudclaim in your tokens, or adjust your token generation to include the correct audience. For example, if your tokens haveaud: "pulsar-cluster-1":# broker.conf token.audience=pulsar-cluster-1 - Why it works: Ensures that the broker only accepts tokens intended for its specific service.
6. Token Revocation: While not strictly "rotation," sometimes tokens are explicitly revoked by the authentication provider. If the Pulsar broker is configured to check revocation lists (e.g., via a JWKS endpoint that includes revocation information) and the token is on that list, it will be rejected.
- Diagnosis: Consult the documentation for your specific authentication provider and Pulsar’s JWT configuration regarding revocation mechanisms. Check if a revocation list is being used and if the token is present.
- Fix: Issue a new token from the authentication provider if the old one was revoked. Ensure Pulsar’s configuration correctly points to the revocation source if applicable.
- Why it works: Enforces security policies by invalidating compromised or no-longer-trusted tokens.
Once these issues are resolved, clients should be able to obtain and renew their JWTs successfully. The next potential hurdle you might encounter is related to authorization policies (authz.conf) if the authenticated user doesn’t have permissions for the requested operations.