Safety model

Every change is an operation

Adding a controller, rebalancing, replacing a broker or upgrading a cluster is an operation: a list of ordered steps stored in PostgreSQL. An operation is planned, running, succeeded, failed or cancelled, and it can be paused between steps. Steps are idempotent, so after a restart of the server the current step simply runs again.

The rules

1. Plan, then confirm. Every operation starts as a dry run that shows each step and each move. Nothing changes until you confirm it.
2. Health gate before every step. No offline partitions, no under-replicated partitions except those being moved, nothing under min.insync.replicas, a quorum leader is present and the voters are caught up.
3. One operation per cluster. A database lock makes sure two operations never change the same cluster at once.
4. One quorum change at a time. A new controller is added and caught up before an old one is removed.
5. Throttled reassignments. Every reassignment is throttled, and the throttles are removed whatever the outcome.
6. Typed confirmation for irreversible actions. Finalizing metadata.version or kraft.version, unregistering nodes and wiping disks ask you to type the name of the cluster.
7. Audit. Who planned, confirmed, paused or cancelled an operation, and the result of every step, stays in the audit log.

Roles

Viewers read everything, including the audit log. Operators change clusters and run operations. Admins also manage users and delete clusters.

Tested on real clusters

The scenarios behind these rules (scaling out and balancing, growing and shrinking the quorum, replacing lost brokers, decommissioning, rolling upgrades and quorum migration) run as end-to-end tests against real Kafka clusters in a lab, in continuous integration.