Six vendors, one API
A provisioning server has one job: a phone asks who it is, and the server tells it. Underneath that, it has six jobs, because we support six manufacturers and no two of them agreed on anything.
Nothing is shared
Start with the file. Four of the six take XML — but four mutually incompatible shapes, with different root elements, different nesting and different conventions for expressing a setting. One of those uses opaque numbered parameter codes rather than names, so the config is a list of numbers mapped to values and the mapping lives in a vendor PDF. The other two take key-value text in two different dialects, and one of those has to be AES-encrypted in transit because the firmware will not accept it otherwise.
Then the directory format differs again — independently of the config format. A vendor whose config is XML does not necessarily use a directory schema resembling its config schema, and none of the six directory formats match each other either.
None of this is standardised. There is no specification you can implement once. There is a document per vendor, of varying quality, sometimes describing the firmware from two releases ago.
Where the abstraction holds
Plenty of it does work. Authentication, identifying the device, loading its configuration, deciding which directory it should see, rate limiting, logging — all of that is genuinely shared. The vendor-specific part is narrower than it first appears: take a resolved set of settings and render it in this vendor's dialect.
So the shape is a common pipeline with a rendering step per manufacturer, which is the obvious design and the right one.
Getting there was itself a goal of the rewrite. The previous code had diverged per vendor much further than the vendors required — the six paths differed in structure, naming and approach as well as in file format, because each had been written or extended at a different time by different people. Pulling them back together so that the only difference is the difference that genuinely exists was as much the point as preserving behaviour was. It is the difference between adding a seventh manufacturer being a day's work and being a project.
The trouble is not the rendering. The trouble is that the settings themselves do not mean the same thing.
Where it doesn't
Consider "line key 3" — the third programmable button down the side of a phone. It sounds like an index into a list. It is not, quite.
On most vendors those keys are numbered from one. On one, they are numbered from zero, so every key on the device is one place out from where the rest of the system thinks it is. That difference is one character in the code and a genuinely confusing afternoon before you find it, because the symptom is not an error — it is a phone where every button does the job of its neighbour.
Then models differ within a vendor. A handset with an expansion module might have forty-eight keys where the base model has a dozen. Code that assumed a single page of keys walks off the end of an array. A related fix: busy-lamp indicators displaying correctly on the first page of keys and incorrectly on every page after, because paging behaviour was one of the things that varied and had been modelled as though it did not.
Individually these are trivial. Several were literally one-character diffs. What they cost was diagnosis, because the abstraction was working perfectly — it was faithfully rendering a correct intention into a dialect where that intention meant something slightly different.
The database ends up describing the hardware
The honest response to this is not more clever code. It is data.
Our device type model carries the number of line keys, the keys per page, the maximum number of pages, whether the model supports hot desking and where its log-in and log-out keys sit, how many directories it supports, whether it can be auto-provisioned at all. A separate table records which features each model supports, because support is per-model and not per-manufacturer.
That schema is essentially a machine-readable description of a hardware catalogue. It looks like over-modelling until you try the alternative, which is conditionals scattered across six rendering paths that each encode a fact about a specific handset. Every quirk you push into data is a quirk that stops being a branch.
It does not eliminate vendor-specific code. It shrinks it to the part that is genuinely about dialect, and moves everything that is about this model's capabilities into rows you can change without a deploy.
What I took from it
You can abstract transport. You cannot abstract semantics. Writing six renderers for six file formats is tedious but tractable — it is mechanical, and you can test it. The cost lands where two vendors use the same word for subtly different things, because there the abstraction compiles, passes review, renders valid output, and is wrong.
Off-by-one is a semantic difference, not a bug. A vendor numbering keys from zero is not incorrect. It is a different convention, and the failure is in the layer that assumed conventions were shared. Treating it as a bug to be patched rather than a fact to be recorded means fixing it repeatedly.
Put hardware facts in data. Anything of the form "this model has N
of X" or "this model supports Y" belongs in a row, not an if. You will
be wrong about the facts — vendors ship new models, firmware adds capabilities — and
you want being wrong to be an update, not a release.
The one-character diffs are the expensive ones. Nothing in the history of this project is less proportionate to its cost than the fixes that changed a single digit. That is what integrating with hardware you do not control looks like: the code is trivial, the knowledge is not, and the knowledge is only obtainable by getting it wrong in front of a real device.
Also in this series
- The hard part was the devices already out there — the rewrite and cutover
- The whitespace was load-bearing — valid XML, truncated anyway
- Nonces don't fit in one process — auth state after horizontal scaling
- Cheaper for the server, dearer for the phone — trading memory for latency
- Everyone checks in at once — fixed intervals and thundering herds
- Writing to both — dual writes during a live migration
- Expected failures are not errors — rolling out tracing and useful logs