The Secret Rotation That Doesn't Wake You Up at 3 AM

Every platform engineer has a version of this story.

Someone files a ticket: “Security wants us to rotate the Kafka passwords.” Easy, right? Update the value in Vault, bounce the service, close the ticket. You schedule it for a quiet Tuesday.

Then at 3 AM your phone lights up. Producers are throwing NOT_ENOUGH_REPLICAS. Half your consumers have fallen out of their groups. Inter-broker replication is throwing authentication errors and partitions are losing leaders. The “quiet Tuesday” is now a Sev-1 with eleven people on a bridge call.

What went wrong? Nothing, technically. You changed a password. That’s exactly the problem.

This article is about why rotating secrets in a stateful, connection-heavy system like Kafka is a genuinely hard distributed-systems problem — and how to do it so that the stream never stops, nobody gets paged, and ideally nobody even notices.

First, find yourself on the ladder. Before the details, here’s the maturity arc the rest of the article unpacks. Locate where you are — it tells you which sections matter most:

Level 0 — plaintext passwords in config files, changed by hand, restart required.

Level 1 — secrets in Vault, rendered to tmpfs, file-based provider, controlled rolling restart.

Level 2 — clients rotate without dropping connections (re-auth + file-backed credentials).

Level 3 — certificate-based, dynamic, auto-reloading (Vault PKI + short TTL + mTLS).

Level 4 — invisible rotation: hourly, automatic, audited, auto-rollback. Nobody notices.

Most teams are at 0 or 1. The goal of this piece is to give you the map to climb.

The real problem isn’t the password. It’s the gap.

Here’s the mental reframe that changes everything:

Secret rotation is not “change the credential.” It’s “manage the window between the moment the identity changes and the moment every actor using it learns the new one — without rejecting a single connection.”

For a stateless web service behind a load balancer, that window is milliseconds and nobody cares. For Kafka, that window is where careers go to die. Because Kafka has three properties that make rotation a contact sport.

1. Authentication happens once, at connection time.

The SASL/SSL handshake runs when the TCP connection opens. After that, the connection lives for minutes or hours, fully trusted. So when you change a credential:

Existing connections don’t care — they already authenticated.
New connections demand the new credential.

This split personality is both your friend and your enemy. It won’t kill live traffic. But the moment any connection drops and reconnects — a deploy, a network blip, a consumer rebalance — it comes back with the old credential and gets rejected. You’ve planted a landmine that goes off on the next reconnect.

2. SCRAM gives you exactly one password per user. No overlap.

A SCRAM-SHA-512 user cannot have both an old and a new password at the same time. This single fact is the hardest knot in the whole problem, and we’ll spend real time on it.

3. The broker is also a client.

Inter-broker replication authenticates too. So changing the broker-internal credential doesn’t just affect “the apps” — it affects the cluster talking to itself. Get this wrong and you break the data plane, not just some consumer.

One piece of good news: In KRaft mode, the controller quorum usually runs on a separate listener. If your SCRAM rotation only touches the data-plane listener, the metadata quorum stays healthy the entire time. And if you’re still on a ZooKeeper-based cluster (plenty of large ones exist), the same comfort applies from a different angle: the inter-broker credential is separate from ZooKeeper’s own auth, so rotating it touches only the broker-to-broker data channel — not your ZK connections. Either way, knowing exactly what’s safe is half of staying calm during a rotation, and it stops you from panic-poking the wrong credential at 3 AM.

The one idea behind every zero-downtime rotation: overlap

Strip away all the tooling and there’s a single principle underneath:

During the transition, the old and new credentials must both be valid, so that no matter which actor presents which identity at which moment, nothing gets rejected.

Telecom people call this make-before-break: establish the new path before you tear down the old one. BGP does it. TLS cert rotation does it. Your rotation strategy is, fundamentally, your answer to one question:

“Does this secret type support overlap? And if it doesn’t, how do I fake it?”

There are only three possible answers:

Native overlap exists → use it (adding a new CA to a truststore, the TTL on a dynamic secret).
No native overlap, but you can fake it with blue-green identities → stand up a new user next to the old one, drain traffic over, retire the old.
Neither → shrink the blast radius (rolling, replication, canary) until the transition window is provably safe.

Now here’s the part most write-ups skip: “Kafka secrets” is not one problem. It’s four problems wearing a trench coat, and each has different physics.

Classify your secrets. They do not rotate the same way.

The first thing I do as a senior engineer is refuse the phrase “rotate the Kafka secrets.” Which secrets? They behave completely differently.

Client SASL credentials (your producers and consumers)

Difficulty: low-to-medium. Lots of actors use these, but they’re all “external” — they don’t touch the cluster’s internal consistency.

The clean paths:

Blue-green users (the bulletproof option): instead of rotating app-producer, create app-producer-v2. Roll the new user out to apps (they pull from Vault too). Once traffic is fully on v2, remove v1 from the ACLs. At no point are both identities invalid. Overlap, guaranteed.
KIP-368 re-authentication (connections.max.reauth.ms): this is Kafka’s unsung hero feature. Set it on the broker and clients are forced to re-authenticate on a schedule without tearing down the TCP connection. If your client reads its credential from a file (the one Vault Agent renders), it picks up the new password on the next re-auth — no reconnect storm. Re-auth fails? The connection closes gracefully and re-establishes. Clean.

Check your clients before you bet on this. KIP-368 shipped in Apache Kafka 2.2 (broker and Java client), so the JVM ecosystem (Spring Kafka, Streams) is fine on any remotely modern version. But many of your producers aren’t Java — they’re built on librdkafka (Python, Go, .NET, C/C++), which added re-auth support later in its 1.x line. Confirm the actual client library and version in your fleet. If you have laggards that don’t support re-auth, you don’t get reconnect-free rotation for them — fall back to the blue-green user strategy for those.

The practical move: feed the credential through a login module that re-reads from a file, not a static sasl.jaas.config string. Now “password changed” becomes “file changed,” and re-auth picks it up automatically.

The inter-broker credential — the hard knot

Difficulty: high. This is the cluster’s nervous system, and single-password SCRAM means no overlap.

Three strategies, in order of maturity:

Level 1 — Controlled rolling restart. Stage the new password in Vault and tmpfs, flip the SCRAM credential in metadata, then restart brokers one at a time. Be precise about what happens here, because the naive mental model (“one broker is down”) is wrong and the wrongness is dangerous. The restarted broker isn’t down — it’s up, healthy, and presenting the new credential, which the metadata now accepts. But the other two brokers are still presenting the old credential, so they can’t open new inter-broker connections to the restarted node. The practical effect: UnderReplicatedPartitions (URP) spikes immediately, and partitions led by the just-restarted broker can see their ISR shrink to one — which means acks=all writes to those partitions may be rejected until the peers catch up. Partitions led by the still-old brokers keep writing fine. This is exactly why the health gate isn’t optional: you wait for URP to return to 0 before restarting the next broker. Restart two in a row without that gate and you can drop below min.insync.replicas and lose writes. Do it gated, ideally during lower traffic, and a healthy cluster rides through with — at worst — a brief, partition-scoped acks=all hiccup, not a cluster outage.
Level 2 — Blue-green inter-broker user. Temporarily run a second superuser and migrate via rolling config. Complex, and Level 1 is usually enough, so this is rare.
Level 3 — Move inter-broker auth to mTLS (the strategic fix). Here’s the senior-level insight: the single-password constraint disappears, because certificates support overlap natively. mTLS certs hot-reload. If inter-broker auth is certificate-based, rotation stops being a “restart” problem and becomes a “cert renewal” problem — which is dramatically cleaner. But mTLS is not a free lunch, so be honest about the bill: you now own a PKI lifecycle (revocation via CRL/OCSP, intermediate rotation, PKCS12 vs JKS quirks), and SSL store hot-reload has been stable only in reasonably recent Kafka versions — validate it on your version before depending on it. Realistically, if you don’t already run Vault PKI, the migration is a multi-month program, not a sprint. The pragmatic path is almost always: start at Level 1 (SCRAM rolling) to stop the bleeding now, then evolve toward mTLS deliberately. Don’t let “mTLS is the right answer” stall you from doing the right first thing this quarter.

TLS certificates — where overlap is generous

Difficulty: medium — but genuinely zero-downtime when done right.

There are two independent overlaps, and the order is everything:

Truststore overlap (the CA): moving to a new CA? Add the new CA cert to every truststore first, keeping the old one. Only after every party trusts both CAs do you rotate the leaf certs. Do it in the wrong order — leaf first — and trust breaks instantly. Outage.
Keystore hot-reload: Kafka can reload its SSL keystore/truststore while running. New connections use the new cert; existing connections are untouched. No restart.

Now combine with Vault PKI: Vault’s PKI engine mints short-lived broker certs (say, 24-72h). Vault Agent renders them to tmpfs and auto-renews at half-TTL. Kafka detects the file change and reloads. The result: certificates rotate continuously, automatically, with zero downtime, and no human ever touches them. This is the summit.

Keystore/truststore passwords — don’t rotate them, design them away

This is the most misunderstood item on the list. A keystore password is the local password that unlocks a certificate file. It has no network meaning; it does not secure the certificate. So:

If it leaked to disk or logs, rotate it (we did — keytool -storepasswd).
But the right long-term move is to take it off the rotation surface entirely: rotate the cert material via Vault PKI and keep the store password static and low-value (in tmpfs, never on disk). Spend your rotation energy on the certificate, not the password.

Secret type	Native overlap?	Zero-downtime method
Client SASL	No (single password)	Blue-green user + KIP-368 re-auth
Inter-broker SASL	No	Rolling restart (replication protects) or move to mTLS
TLS certificate	Yes (truststore + hot-reload)	Vault PKI + short TTL + auto-reload
Store password	—	Rotate the cert, take the password off the surface

The architecture that makes “smooth” possible

A rotation strategy is only as graceful as the infrastructure under it. The reference architecture I build has four layers:

1. Vault is the single source of truth. All credentials live only in Vault. KV for static secrets; dynamic secrets (PKI, database engine) wherever possible — because a dynamic secret’s TTL gives you overlap for free.

2. Vault Agent on every node (or Sidecar/CSI on Kubernetes). The agent authenticates via AppRole / Kubernetes auth and renders secrets to tmpfs (RAM). Never to disk; gone on reboot. It auto-renews at half-TTL. One detail worth knowing so you don’t invent a problem that isn’t there: Vault Agent’s templating writes atomically by default — it renders to a temp file and rename()s it into place, so a reader never sees a half-written secret. And because Kafka’s DirectoryConfigProvider resolves values at startup/reconfiguration rather than continuously polling the file, there’s no torn-read race in this design. (If you ever build something that does watch the secret file live, that atomic rename is exactly what saves you — so don’t disable it.)

3. The app/broker reads the secret from a file, re-readably. Not a static environment variable — a file-based config provider (Kafka’s DirectoryConfigProvider) or a login callback that re-reads. Re-readability is the precondition for zero downtime. Environment variables are a dead end for rotation: they require a process restart and they leak into child processes.

4. Turn on the hero features: connections.max.reauth.ms (clients rotate without reconnecting), SSL store dynamic reload (certs rotate without restart), and inter-broker mTLS where you can (it dissolves the hardest knot).

A scar I’ll share: it’s tempting to put a systemctl restart kafka exec on the Vault Agent template so secrets “just apply.” Don’t. If all three nodes’ agents see the new secret at roughly the same time and restart at roughly the same time, your RF=3 / min.insync=2 cluster goes down together and you get the exact outage you were trying to avoid. Separate rendering (automatic, continuous) from rollout (controlled, rolling). That separation is the entire difference between “automatic rotation” and “automatic outage.”

Here’s the same idea as a timeline — the backbone of this whole article in one picture:

 RENDER  (automatic, continuous, harmless)        ROLLOUT (controlled, gated, one node at a time)
 ────────────────────────────────────────►        ─────────────────────────────────────────────►
 Vault KV updated                                  flip metadata / signal node
   │                                                 │
   ▼                                                 ▼
 Agent renders new secret → tmpfs (all nodes)      node-1 ──[health gate: URP=0?]──► node-2 ──[gate]──► node-3
   │  (nothing applied yet — cluster untouched)      │            │                                    │
   └─ new value sits ready, old value still live ◄───┘     verify each before continuing     ◄─ rollback if a gate trips
        ^^^ this is your overlap window ^^^

The left side can happen a hundred times a day and never touch a connection. The right side is the only part that carries risk — so it’s the only part you gate, canary, and make reversible.

How I actually orchestrate it

Here’s the choreography I run in the field. The key is putting every step behind a health gate and staying reversible.

Step 0 — Prerequisites (weeks before): Are clients reading the credential from a file with re-auth enabled? If not, fix that first. Are the dashboards ready (auth failure rate, under-replicated partitions, request error rate, consumer lag)? Has the rotation been rehearsed in staging on the same topology — a real game day?

Step 1 — Prepare (build the overlap): Write the new value to Vault (via a file, never on the command line). Refresh the agents → the new value is staged in tmpfs. No rollout yet. Blue-green? Create the new user and ACLs, leave the old one valid.

Step 2 — Verify the new identity is accepted: Open a test connection with the new credential. If it fails, this is your stop point — you haven’t rolled out anything, the system is intact.

Step 3 — Canary: Roll out on a single broker (or one client deployment). Watch the health gate. Clean for 5-10 minutes? Continue.

Step 4 — Rolling rollout, gated: Go node by node. After each one: under-replicated = 0? ISR full? Auth-failure rate back to baseline? Don’t move to the next node until the gate is green. For overlap-less cases like inter-broker SCRAM, only one node is ever “in transition” — replication covers you.

Step 5 — Retire the old: Blue-green? Remove the old user/ACLs. SCRAM flip? The old password is already gone from metadata. Redact old values from logs; shred old backups.

Step 6 — Automatic rollback: If a health gate trips, the orchestration must stop and roll back on its own. A human can approve each gate, but the abort must be automatic. Deciding by hand, half-rotated, at 3 AM, is the mother of all incidents.

Rollback has a SCRAM-shaped catch — design for it up front. When you flip a SCRAM credential with kafka-configs --alter, the old password is overwritten and gone from metadata. So “roll back” isn’t free: it means writing the previous password back into metadata. This is exactly why you keep credentials in versioned KV (Vault KV v2) — your rollback step reads the prior version (vault kv get -version=N) and re-runs --alter with it. If you skip this, “rollback” silently degrades into “hope someone still knows the old password,” which is not a plan. (Blue-green users dodge this entirely: rollback is just pointing traffic back at the still-present old user.)

Observability is the real safety net. Four metrics to watch during the window: (1) failed-authentication rate, (2) under-replicated/offline partition count, (3) produce/fetch error rate — especially acks=all, (4) consumer-lag derivative. Wire them to SLO thresholds scoped to the rotation window.

Running it like a senior engineer: process and governance

The mechanism is half the job. The other half is turning it into a system that survives human error and scale.

Make rotation a feature, not an event. Manual, once-a-year, panic-driven rotation is a failure mode. The goal is for rotation to be so automatic and safe that it happens without anyone knowing. Short TTLs + dynamic secrets + auto-reload get you there. The right north star isn’t “rotate more often because it’s safer” — it’s “rotation is so cheap we can do it constantly.”

Take secret-zero seriously. The identity Vault Agent uses to log in to Vault (AppRole secret_id, a Kubernetes service account) is the weakest link in the chain. Short TTLs, CIDR binding, response-wrapping, and platform-native identity (K8s/AWS IAM auth) where possible. Plan the rotation of that identity too.

Least privilege and blast radius. Each node reads only the path it needs (secret/data/kafka/*, read-only). If a node is compromised, the attacker sees that node’s secrets — not the whole universe.

Audit and break-glass. Keep Vault’s audit log on — who read what, when. Define a break-glass procedure for when the automation itself fails, and rehearse it.

The operation itself is a leak source. From experience: the rotation commands are the most common leak point. Pass secrets via files (mode 0600), via stdin prompts (printf | keytool), via --command-config — never as command-line arguments, never through sudo (it lands in auth.log). Once a secret has touched disk or a log, consider it burned and rotate it.

An untested rotation is not a rotation. Run game days in staging on a prod-shaped topology with simulated traffic. See the min.insync behavior, confirm re-auth actually works on clients, confirm rollback works. Production should not be the first time you try it.

Field traps (paid for in pages)

A trailing newline in the rendered secret file. DirectoryConfigProvider takes the file content verbatim; a trailing \n corrupts the password and you get “invalid credentials” while you search everywhere except the right place. The fix: use trim markers in the Vault Agent template — {{- with secret "..." -}}{{ .Data.data.password }}{{- end -}} (the dashes eat surrounding whitespace and newlines) — and then prove it with od -c <file> or tail -c1 <file> | xxd: the last byte must be a password character, not 0a. I’ve watched a team burn an afternoon on a “wrong password” that was a single newline.
The Vault Agent template that restarts the whole cluster at once. (See the scar above.) Automatic render ≠ automatic rollout.
Confusing the control/quorum plane with the data plane. Know which listener uses which auth before you rotate; knowing the quorum is safe keeps you calm.
Rotating the leaf cert before distributing the new CA. Instant trust break. Always truststore-first.
Trusting environment variables. Dead end for rotation; forces a restart and leaks into child processes.
OOM heap dumps and core dumps. Secrets are plaintext in RAM; a heap or core dump writes them to disk. LimitCORE=0, tight permissions on the dump directory, disable OOM dumps if you can.
“We rotated, but the old value still works somewhere.” Prove the old credential is actually rejected with a negative test — don’t just confirm the new one works. Concretely: point a client at the cluster using the old JAAS config and run a read-only call, e.g. kafka-broker-api-versions.sh --bootstrap-server <broker>:9092 \--command-config old-creds.properties. If it authenticates, the rotation failed — the old credential is still alive somewhere. Rotation is only complete when the old value is dead and you’ve watched it die.

A maturity ladder

Find your rung and aim one step up:

Level 0 — Static & manual: passwords in plaintext config files, changed by hand, restart required. Where most teams start; the most fragile.
Level 1 — Centralized & controlled: secrets in Vault, rendered to tmpfs by the agent, file-based provider, controlled rolling restart. No plaintext on persistent disk.
Level 2 — Reconnect-free client rotation: KIP-368 re-auth + a login module that re-reads from file; client credentials rotate without dropping connections.
Level 3 — Certificate-based & dynamic: Vault PKI + short TTL + SSL hot-reload; inter-broker mTLS. Certificates rotate automatically, continuously, with zero downtime.
Level 4 — Invisible rotation: rotation is so frequent, cheap, and automatic that it’s not an “event.” Secrets turn over hourly, nobody notices, audit sees everything, rollback is automatic.

Closing

Zero-downtime secret rotation isn’t magic. It’s disciplined overlap engineering. It fits in three sentences:

Classify the secret — each type has different physics; you can’t rotate a single-password SCRAM credential the way you rotate a certificate.
Build the overlap, or fake it — old and new must never be invalid at the same time; if they can’t overlap, shrink the blast radius with replication and rolling.
Separate render from rollout, gate every step on health, stay reversible — automation comes with a safety net, or it’s just a faster incident.

And the real senior-level insight is this: the best rotation is the one nobody notices. Your goal isn’t to “successfully change the password.” It’s to make rotation so routine and safe that the stream never breaks, no one holds a pager, and security stops being a cost and becomes a quiet property of the system breathing normally.

When you get there, you’ll have rotated the credentials a hundred times — and you’ll have slept through every single one.