Helm charts are the standard way to deploy Pulsar on Kubernetes, but they’re more than just a deployment tool; they’re your blueprint for a resilient, scalable Pulsar cluster.
Let’s see Pulsar in action, managed by Helm. Imagine this: you’ve just applied your Pulsar Helm chart.
helm install my-pulsar pulsar/pulsar --namespace pulsar
Seconds later, kubectl get pods -n pulsar shows a fleet of pods coming online: bookkeeper-0, bookkeeper-1, zookeeper-0, zookeeper-1, zookeeper-2, pulsar-broker-0, pulsar-broker-1, and pulsar-manager. This isn’t magic; Helm orchestrated the creation of Kubernetes Deployments, StatefulSets, Services, and ConfigMaps based on the chart’s templates and your values.yaml overrides.
The core problem Pulsar on Kubernetes solves is providing a distributed messaging system that can dynamically scale and self-heal within the ephemeral nature of containers. Pulsar itself is composed of several key components: ZooKeeper for metadata and coordination, BookKeeper for persistent message storage, and Pulsar brokers for serving client requests and managing topics. Each of these has specific Kubernetes resources defined in the Helm chart.
The Helm chart abstracts away the complexity of setting up these components. It defines:
- ZooKeeper: Typically deployed as a StatefulSet to ensure stable network identifiers and persistent storage for each ZooKeeper node. The chart configures quorum size, data directories, and resource requests/limits.
- BookKeeper: Also a StatefulSet, mirroring ZooKeeper’s need for stable identities. The chart specifies the number of bookies, their storage configuration (using
persistentVolumeClaimTemplates), and resource allocation. - Pulsar Brokers: Deployed as a Deployment or StatefulSet, depending on the chart version and configuration. This manages the stateless (or mostly stateless) broker instances responsible for topic lookups and client connections.
- Pulsar Manager: A web UI for managing Pulsar resources, deployed as a standard Deployment.
- Services: Kubernetes Services are created for each component (ZooKeeper, BookKeeper, Brokers) to provide stable network endpoints, enabling inter-component communication and client access.
- ConfigMaps: These hold the Pulsar configuration files (
server.conf,broker.conf, etc.), allowing the chart to inject dynamic settings based on yourvalues.yaml.
The exact levers you control are primarily through the values.yaml file. Want more broker capacity? Change broker.numBrokerReplicas: 3 to broker.numBrokerReplicas: 5. Need faster storage for BookKeeper? You’ll adjust bookkeeper.volume.storageClassName: "fast-ssd" and potentially bookkeeper.volume.size: "100Gi". Resource limits and requests are crucial for stable operation: zookeeper.resources.limits.cpu: "500m", zookeeper.resources.limits.memory: "512Mi".
Here’s a peek at a simplified values.yaml snippet:
global:
imageRegistry: docker.io
imagePullPolicy: IfNotPresent
zookeeper:
replicas: 3
resources:
requests:
cpu: 200m
memory: 512Mi
limits:
cpu: 500m
memory: 1Gi
persistence:
enabled: true
storageClass: "standard"
size: 10Gi
bookkeeper:
replicas: 3
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: 1000m
memory: 2Gi
volume:
enabled: true
storageClass: "standard"
size: 100Gi
broker:
replicas: 2
resources:
requests:
cpu: 1000m
memory: 2Gi
limits:
cpu: 2000m
memory: 4Gi
pulsarManager:
enabled: true
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 200m
memory: 512Mi
The chart’s power lies in its templating engine, which generates Kubernetes manifests dynamically. For instance, the zookeeper.conf is templated, incorporating the number of replicas and node names derived from the StatefulSet’s generated pod.name. This allows you to scale ZooKeeper by simply changing zookeeper.replicas and re-applying the chart.
One aspect often overlooked is how the chart manages Pulsar’s internal communication endpoints. Services like my-pulsar-zookeeper-headless and my-pulsar-bookkeeper-headless are crucial for StatefulSets, providing stable DNS entries for each pod. The broker service, my-pulsar-broker, is what clients connect to. The chart ensures these services are correctly configured with the right selectors to target the respective pods. When you scale brokers, the service automatically picks up the new instances, provided they are ready.
When you scale a component, say brokers, by changing broker.replicas and running helm upgrade my-pulsar pulsar/pulsar -n pulsar -f values.yaml, Helm doesn’t just create new pods. It updates the Kubernetes Deployment/StatefulSet definition. Kubernetes then notices the desired replica count is higher and starts provisioning new pods. As these new broker pods start up, they read their configuration from the ConfigMap, discover the existing ZooKeeper and BookKeeper endpoints (via their Service), and join the cluster. The service my-pulsar-broker will then start load-balancing traffic to these new instances.
The next step after getting a stable Pulsar cluster running is often configuring Pulsar functions or setting up advanced authentication.