Brokerium

October 19, 2026 · Bulat Valiev

Changing the KRaft controller quorum without downtime

The controllers of a KRaft cluster form a Raft quorum that stores the cluster metadata. Changing that quorum, for example growing it from three to five controllers or replacing a host, used to mean rewriting controller.quorum.voters everywhere and restarting. With a dynamic quorum it is an online operation, as long as the steps happen in the right order.

Static and dynamic quorums

A static quorum lists its voters in controller.quorum.voters on every node; it uses kraft.version 0. A dynamic quorum stores the voters in the metadata log itself: nodes only need controller.quorum.bootstrap.servers to find the quorum, and voters are added and removed with the Admin API. It uses kraft.version 1.

Adding a controller

  1. Format the new controller’s storage with --no-initial-controllers and start it. It joins as an observer and replicates the metadata log.
  2. Wait until it has caught up with the leader (kafka-metadata-quorum.sh describe --replication shows its lag).
  3. Promote it to a voter, from the new node:
Terminal window
kafka-metadata-quorum.sh --bootstrap-controller controller-1:9093 \
  --command-config server.properties add-controller

Removing a controller

Remove it from the voters first, with its node id and directory id, and only then stop it:

Terminal window
kafka-metadata-quorum.sh --bootstrap-controller controller-1:9093 \
  remove-controller --controller-id 4 --controller-directory-id <directory id>

The rules that keep the cluster available

  • One change at a time. Never add and remove in parallel.
  • Add before you remove. When replacing a controller, the new one becomes a voter before the old one leaves, so the quorum never shrinks below what it can tolerate.
  • Only caught-up voters. A voter that is far behind can stall the quorum if another controller fails.
  • Keep an odd number of voters. Three tolerate one failure, five tolerate two.

From a static to a dynamic quorum

On recent Kafka versions a static cluster can upgrade kraft.version to 1 in place:

Terminal window
kafka-features.sh --bootstrap-server broker-1:9092 upgrade --feature kraft.version=1

The upgrade cannot be undone. Afterwards the voters are stored with their directory ids, and the nodes can switch from controller.quorum.voters to controller.quorum.bootstrap.servers.

How Brokerium does it

Brokerium runs these steps as operations: it plans them, asks for confirmation, checks before every step that the quorum has a leader and that the voters are caught up, allows only one quorum change at a time and asks you to type the cluster name before the irreversible kraft.version upgrade. The exact commands are in the controller quorum runbook and the static to dynamic quorum runbook.

All articles