Migrating Pulsar from ZooKeeper to the native BookKeeper metadata store (Oxia) doesn’t just change where your metadata lives; it fundamentally alters how Pulsar coordinates and scales.
Let’s see it in action. Imagine a Pulsar cluster with a few brokers, a BookKeeper ensemble, and ZooKeeper handling all the metadata. Clients connect, topics are created, and everything hums along. Now, we’re going to switch that metadata source to Oxia, which is built into BookKeeper itself.
Here’s a simplified view of what happens during a migration. We have our existing Pulsar cluster, and we want to move its metadata from ZooKeeper to BookKeeper’s own metadata layer.
# Example Pulsar Broker Configuration (pre-migration)
listeners: pulsar://broker-0.pulsar.service.consul:6650
brokerServicePort: 6650
zookeeperServers: zk-0.zk.service.consul:2181,zk-1.zk.service.consul:2181,zk-2.zk.service.consul:2181
globalZookeeperServers: zk-0.zk.service.consul:2181,zk-1.zk.service.consul:2181,zk-2.zk.service.consul:2181
After the migration, the configuration will look like this:
# Example Pulsar Broker Configuration (post-migration)
listeners: pulsar://broker-0.pulsar.service.consul:6650
brokerServicePort: 6650
# zookeeperServers and globalZookeeperServers are removed
# The new metadata store is managed by BookKeeper itself
The core problem Oxia solves is the operational overhead and potential bottleneck of managing a separate ZooKeeper ensemble for Pulsar metadata. ZooKeeper, while robust, is an external dependency. It requires its own scaling, monitoring, and maintenance. By integrating metadata storage directly into BookKeeper, Pulsar streamlines its architecture, making it more self-contained and easier to manage.
Internally, Oxia leverages BookKeeper’s ledger and entry mechanisms to store Pulsar’s metadata. Instead of writing metadata to ZooKeeper nodes, Pulsar brokers and BookKeeper itself write metadata entries to BookKeeper ledgers. These ledgers are then managed by the BookKeeper ensemble, providing fault tolerance and durability for the metadata. This means your metadata is now protected by the same replication and quorum mechanisms that protect your message data.
The primary levers you control are related to the BookKeeper ensemble itself and the Pulsar configuration.
- BookKeeper Ensemble Configuration: Oxia relies on the BookKeeper ensemble. Ensure your BookKeeper cluster is healthy and adequately provisioned. The
bookkeeper.conffile will have settings related to metadata storage, though many of these are managed internally by Oxia. Key parameters for the BookKeeper ensemble itself includezkLedgersPath(where BookKeeper stores its own internal metadata, which Oxia can potentially replace later),zkMetadataServiceUri(which Oxia will eventually supersede), andensembleSize,writeQuorum, andackQuorumfor the BookKeeper cluster. - Pulsar Broker Configuration: This is where the explicit switch happens. You’ll remove
zookeeperServersandglobalZookeeperServersfrombroker.confandoperators.conf(if using Pulsar Operator) and introducemetadataStoreconfiguration pointing to BookKeeper. The critical change ismetadataStore.provider=bookkeeperandmetadataStore.uri=bk://<bookie-address>:2181(where the2181is the ZK port BookKeeper still uses for its own service discovery initially, but the metadata itself is no longer stored there). A more direct configuration for Oxia might look likemetadataStore.uri=bk://bookkeeper-0:2181,bookkeeper-1:2181,bookkeeper-2:2181.
The migration process itself typically involves:
- Configuration Update: Modifying Pulsar broker and BookKeeper configurations to point to the new metadata store.
- Metadata Copying: A utility or process to copy existing metadata from ZooKeeper to the new BookKeeper-based store.
- Service Restart: Restarting Pulsar brokers and potentially BookKeeper instances to pick up the new configuration and metadata source.
The most surprising true thing about Oxia is that it can operate in a transitional state where BookKeeper itself still uses ZooKeeper for its own internal service discovery and ledger management, while Pulsar’s metadata has already moved to Oxia. This allows for a phased migration, reducing the risk of a single cutover event.
Once you’ve successfully migrated to Oxia, the next logical step is to explore consolidating BookKeeper’s own metadata storage away from ZooKeeper, leveraging Oxia even further for a fully ZooKeeper-free Pulsar deployment.