Nonces don't fit in one process

Most of what broke after we moved our provisioning server from Python to Kotlin broke because of the devices. This one broke because of us — specifically because the new service ran as several instances behind a load balancer, and the old one had been a single process.

Nothing about the protocol changed. Nothing about the devices changed. The deployment topology changed, and that was enough.

How digest auth works

One manufacturer's handsets authenticate with HTTP digest rather than basic auth. The exchange goes:

1. The device requests its config with no credentials.
2. The server responds 401 with a challenge containing a nonce — a one-time value the server has just generated.
3. The device hashes its username, password and that nonce together, and retries with the result.
4. The server computes the same hash using the nonce it issued and compares.

The nonce is what stops the exchange being replayable: the password never crosses the wire, and a captured response is useless against a different challenge. It only works if the server can remember, between two separate HTTP requests, which nonce it gave to which device.

Why one process made that invisible

On a single process, remembering is free. Put the nonce in a dictionary keyed by device, look it up on the retry, done. It is barely a design decision — it is the obvious thing, it works, and nothing about it announces that it depends on there being exactly one of you.

Run three instances behind a load balancer and the assumption quietly stops holding. The device's first request lands on instance A, which generates a nonce and stores it in its own memory. The retry — a separate TCP connection, load-balanced independently — lands on instance B, which has never heard of that nonce. B cannot verify the hash, so it rejects the request and issues a challenge of its own. The device retries against that, lands on instance C, and so on.

With three instances a device authenticates when it gets lucky and lands twice in a row on the same one. That is the worst kind of failure: not "broken", just intermittent, self-resolving on retry, impossible to reproduce on a single local instance, and perfectly healthy in every test that runs against one process.

The fix is boring, which is the point

Move the nonce out of process memory and into shared storage — Redis, keyed by device — so that whichever instance handles the retry can look up the challenge whichever instance issued it.

// before: implicit, per-instance, invisible
nonces[macAddress] = nonce

// after: explicit, shared, has a lifetime
securityService.setNonce(macAddress, nonce)

That is genuinely all it is. The interesting part is not the fix but that the code needing it looked completely correct, had presumably worked for years, and contained no clue that it would stop working. There is no comment to write on a dictionary assignment that says "only valid while this process is the only one".

Worth noting too: once the nonce is in shared storage it acquires properties the local version never had to think about. It needs an expiry, because entries are no longer discarded when a process restarts. It needs a key namespace that cannot collide. And a challenge is now a write to a network service on a path that previously touched nothing but memory.

What I took from it

Horizontal scaling converts implicit local state into explicit shared state — and it does not warn you which state that is. Anything a request stores expecting the next request to find it is a candidate: session data, rate-limit counters, in-flight locks, caches used for correctness rather than speed, and auth challenges. The migration exposes them one at a time, in production, usually as something intermittent.

Multi-step protocols are where this hurts most. A single-request endpoint does not care which instance serves it. A challenge-response exchange spans two requests by definition, so it carries an implicit assumption about instance affinity that nobody wrote down because, on one process, it was not an assumption.

Intermittent auth failures are a topology smell. If something authenticates on the second or third attempt and you cannot reproduce it locally, stop reading the auth code and count how many instances are running. Working "most of the time" in a way that correlates with instance count is not a race condition in your logic — it is state living in the wrong place.

Sticky sessions would also have "fixed" this, and I am glad we did not reach for them. Pinning devices to instances would have hidden the design problem behind a load balancer setting, and left it to resurface the next time an instance was replaced.