Redpanda rolling upgrades are designed to be zero-downtime, but only if you understand that the cluster’s consensus mechanism is the single point of control and the bottleneck for state changes.
Let’s see this in action. Imagine a three-node Redpanda cluster. We want to upgrade redpanda-0 first.
# On the machine running redpanda-0
sudo systemctl stop redpanda
# Perform upgrade steps (e.g., apt update && apt upgrade redpanda, or docker pull, etc.)
# Verify the new binary/image is in place
redpanda --version
# Start Redpanda
sudo systemctl start redpanda
While redpanda-0 is restarting, the other two nodes (redpanda-1 and redpanda-2) continue to serve traffic. They form a quorum and can still make progress on raft operations. Once redpanda-0 rejoins, it will catch up on any missed log entries by streaming them from the other nodes. The key here is that the majority of nodes must agree on any leadership change or partition state update. As long as a majority (2 out of 3 in this case) is running the same or a compatible version, the cluster remains available.
The mental model to hold onto is that Redpanda, like Kafka, is built on the Raft consensus algorithm. Each partition leader uses Raft to replicate its log to followers. An upgrade involves replacing the binary or container image on each node. The rolling upgrade process leverages the fact that Raft can tolerate a minority of nodes being unavailable or temporarily out of sync, as long as a quorum (majority) can still be established.
When you stop a node, the remaining nodes might elect a new leader for the partitions that were led by the stopped node. If a new leader is elected, the other nodes will follow it. Once the original node restarts and rejoins the cluster, it will be elected as a follower for those partitions and will stream the missing log entries from the new leader to catch up. This log catching-up process is crucial and happens in the background without impacting client availability. The cluster only truly halts if it cannot form a quorum for leadership or state updates.
The most counterintuitive part of this process is realizing how much state change can occur while a node is down. Clients might be writing to a partition, and the leader for that partition might be on the node you just stopped. However, Raft’s design allows for a new leader to be elected immediately from the remaining healthy nodes. This new leader then coordinates further replication. The stopped node, upon restart, doesn’t need to re-lead; it simply needs to catch up by becoming a follower and ingesting the replicated log entries it missed. This distinction between leadership election and log synchronization is what enables zero downtime.
The next hurdle in advanced Redpanda operations is understanding how to manage cross-datacenter replication (using Kafka MirrorMaker 2 or Redpanda’s built-in replication) during an upgrade, as its availability is dependent on the availability of both clusters.