Cheaper for the server, dearer for the phone

A desk phone's directory arrives as a file. The phone asks for it, the server builds it, the phone downloads and stores the whole thing. That model is simple and it works right up until somebody has a directory with six thousand people in it.

This is a story about making something meaningfully better for the server and, in the same change, meaningfully worse for a handful of devices — and about the fact that once you have done that, the least capable client in the estate decides where your limit is.

What the split changed

In the old architecture the provisioning server and the directory data lived together. Building a directory meant querying a database the same process could reach, loading the contacts, and rendering them. One query, one in-memory list, done.

The rewrite split those apart. Directory data belongs to a different service now, and the provisioning server asks for it over gRPC, a page at a time, assembling the result as it goes.

That is a genuine improvement at the platform level, and not for abstract architectural-purity reasons. The old shape meant a single process could be asked to hold an entire directory in memory, and with enough directories being built at once by enough accounts, the worst case is a service holding an enormous number of contact records simultaneously. Paging bounds that. No single request can balloon, and memory use stops being a function of your largest customer.

What it cost

Every page is now a network round trip to another service, with serialisation at both ends, instead of a row read from a local result set. For a small directory that is irrelevant — a page or two, a few milliseconds. For a six-thousand-contact directory it is a lot of round trips, and the time to assemble the response scales with the size of the directory in a way it previously did not.

The obvious mitigation helps and we did it: the directory build fetches two independent things — the account's own users and the directory's contacts — and those were being fetched one after the other. Running them concurrently rather than sequentially takes a chunk out of the total.

But that is a constant-factor improvement to something that grows linearly. It buys headroom. It does not change the shape of the curve.

The slowest client sets the limit

Eventually a customer with about six thousand contacts reported that their directory would not load. The detail that makes this worth writing about is that it failed on one model. Other handsets from the same manufacturer, on the same account, with the same configuration and the same directory, were fine.

Nothing was wrong with the server's response. It was correct, complete, and eventually delivered. The failing model simply gave up waiting sooner than its siblings did — a shorter timeout in firmware we do not control and cannot configure.

That is the uncomfortable part of trading latency for resource efficiency when your clients are devices. If you make a server-side saving and pay for it in response time, you have not removed the cost. You have moved it onto the clients, and you now inherit the tolerance of the least patient one. Every model, every firmware revision, every timeout somebody chose years ago is now part of your performance budget.

You cannot fix that by being faster, because you do not know what the limit is. It is not published. You discover it when a customer large enough to cross it complains.

Stop sending the directory

The fix was not a faster download. It was to stop downloading.

Most of these handsets also speak LDAP. Instead of fetching a file containing every contact, the phone can open a connection and search: the user types three letters, the phone sends a query, the server returns the matches. So the provisioning server gained an embedded LDAP listener that serves a virtual directory backed by the same contact data.

The property that matters is that the response size is now bounded by construction. A search returns at most a fixed number of entries — a few dozen — regardless of how many contacts exist behind it and regardless of what the phone asks for. A six-thousand-contact directory and a sixty-contact directory produce responses of the same order. The size of the directory stops being a variable in whether the feature works.

Queries are prefix matches rather than substring searches, which keeps them index-friendly and matches how people actually search a phone directory — you type the start of a name, not the middle of one.

Putting the listener inside the provisioning server rather than standing up a separate directory service was deliberate. That service already knows how to identify a device, work out which account it belongs to and decide which directory it should be allowed to see. All of that is needed to answer an LDAP search safely, and duplicating it elsewhere would have meant two implementations of the same access rules.

What I took from it

Efficiency gains that cost latency are transfers, not savings. The work did not disappear when we bounded the server's memory use; it turned into waiting, and the waiting is done by somebody else. That is usually a good trade — but it is a trade, and it is worth naming as one rather than recording it as a pure win.

Your timeout budget is set by clients you cannot see. With browsers you can measure and iterate. With an estate of hardware spanning years of firmware, the binding constraint is undocumented, varies by model, and only reveals itself in production at a customer large enough to hit it.

When the size of the payload is the problem, optimise the interaction, not the payload. We could have kept making directory assembly faster — more parallelism, caching, bigger pages — and every increment would have moved the failure threshold slightly further out while leaving the shape of the problem untouched. Switching from "send everything" to "answer questions" removes the variable instead of shrinking it.

Two identical-looking symptoms, six months apart, had nothing in common. Earlier in the year, large directories failed on specific handset models because of a whitespace change and a fixed-size line buffer. This time, large directories failed on a specific handset model because of assembly latency and a firmware timeout. Same shape of report, same class of customer, entirely unrelated causes. It is a good reminder not to let a previous diagnosis do your thinking for you.