Everyone checks in at once

Desk phones ask a provisioning server for their configuration on a timer. Set that timer to a day and each phone checks in once a day, which sounds like it spreads the load across twenty-four hours. It does not. It spreads the load across however many distinct moments the phones happened to boot at — and a surprising number of them booted at the same moment.

A site loses power and comes back. A firmware push goes out. Somebody applies a profile change across an account. In each case a block of phones restarts together, starts its timer together, and then — because the interval is identical for all of them — stays in lockstep indefinitely. A daily interval does not decay into a smooth distribution. It preserves whatever clustering existed at boot, forever.

Three herds, not one

The provisioning check-in is the obvious one, but it is not the only periodic timer a provisioning server hands out. It also sets how often the phone refreshes its SIP registration, and how often it renews the subscriptions behind busy-lamp indicators — the little lights showing whether a colleague is on a call.

All three are configured values. All three were fixed. So a cluster of phones that booted together would hit the provisioning server together, re-register together, and renew their presence subscriptions together.

The obvious fix is wrong

The textbook answer to a thundering herd is jitter: add a random offset so the herd spreads out. Generate a random number per request, subtract it from the interval, done.

That would have been a serious mistake here, and the reason is specific to configuring hardware rather than calling an API.

A phone does not re-apply its whole configuration on every check-in. It compares what the server sends against what it currently has and applies the differences. For most parameters that is harmless. For some — one vendor's device-level settings in particular — changing a value triggers a reboot so the phone can pick it up.

If the interval is randomly generated on each request, then the value differs every time the phone asks. The phone sees a changed setting, applies it, and reboots. Then it boots, checks in, receives another new random interval, and reboots again. The obvious fix for "the fleet calls in together" would have been "the fleet reboots continuously".

Deterministic spread

The offset has to be stable per device: different from its neighbours, identical every time that device asks. So it is derived from a hash of the device's MAC address rather than drawn from a random source.

// same device -> same offset, every time
val offset = hash(macAddress + timerSalt) % SPREAD_WINDOW
val interval = configuredInterval - offset

Each phone gets its own point in the window and keeps it, so the generated config is byte-identical between check-ins and nothing is ever seen as changed. The herd breaks apart and nothing reboots.

Two details in that line matter more than they look:

Each timer gets its own salt. If registration and subscription refresh share an offset, then every phone renews both at the same instant — you have spread the fleet out but bunched each device's own traffic into a spike. Different salts per timer put a given phone's registration refresh and its subscription renewal at different points in the hour.

The offset is only ever subtracted. Some of these intervals have protocol or vendor ceilings — a subscription period of 3600 seconds means 3600, not "around 3600". Adding a random offset could push a device over the limit. Subtracting keeps every value inside its bound, at the cost of making the mean slightly shorter: an hour becomes fifty to sixty minutes, averaging fifty-five. Slightly more traffic overall, in exchange for never exceeding a ceiling. That is the right trade.

There is a floor as well, so that a profile asking for an aggressively short interval cannot be spread down into something pathological.

What I took from it

A fixed interval is a synchronisation primitive. It does not distribute load; it preserves whatever distribution existed when the clients started, and every mass restart re-clusters them. If you set a uniform interval across a fleet, you have not scheduled work evenly — you have scheduled it in lumps, and the lumps are wherever the last outage put them.

Jitter has to be deterministic when the client diffs. Randomness per request is fine when the response is consumed and discarded. When the client compares this response against the last one and acts on the difference, a value that changes every time is a value that triggers the action every time. Hash the client identity instead; you get the same spread and a stable answer.

Think about what you are spreading, not just that you are spreading. Sharing one offset across several timers moves the problem from between devices to within a device. It is the same mistake at a smaller scale, and easier to miss.