The whitespace was load-bearing

On a Wednesday morning in March, support started taking calls. Desk phone directories had stopped working — not for one customer, but across an entire manufacturer's range at once. The report that landed with us was unusually clear about how strange it looked:

Since the start of today we have had a large influx of calls and tickets regarding issues with phonebook contacts… This even affects devices that are on the most up to date firmware for that device. Every customer that reported this states it was working fine in the past, including yesterday.

Every part of that is the wrong shape for a normal bug. Not one model — all of them. Not old firmware — current firmware too. Not a gradual degradation — working yesterday, broken today. When a failure arrives evenly across a whole device family overnight, it is almost never the devices. Something we sent them changed.

What we had changed

Two days earlier I had merged a tidy-up. Our provisioning server builds XML documents that handsets fetch on boot — configuration, line keys, directories. Some of that XML was being generated with the indentation and line breaks you would expect from a document builder, and a handful of parsers were unhappy with the leading whitespace. So we normalised it with a small helper:

fun cleanXml(xml: String) =
    xml.filter { it != '\n' && it != '\r' && it != '\t' }
       .trim()

Strip the newlines, carriage returns and tabs, trim the ends, and every document comes out uniform. It fixed the display problem it was meant to fix. It is also about as innocuous a change as it is possible to write.

Both documents were valid

This is the part worth sitting with. Whitespace between elements is insignificant in XML. A pretty-printed document and the same document collapsed onto one line are equivalent — same elements, same attributes, same text content, same tree. Any conforming parser produces an identical result from both.

So there was nothing to catch. No schema violation, no malformed tag, no encoding problem. Feed either version to any XML library you like and it parses cleanly. The document we started sending was, by every measure a validator can express, the same document we had been sending before.

The handsets were not running an XML parser

Or rather, not only one. Before anything reached a parser, the firmware on the affected models read the HTTP response line by line, into a fixed-size buffer. Lines longer than that buffer were truncated, and the remainder of the line discarded.

That behaviour is invisible while documents contain newlines, because every line is short. Remove the newlines and the entire document becomes a single line — and for a directory, the length of that line scales with the number of contacts. A customer with a handful of entries stayed under the limit and kept working. A customer with a real company directory produced one very long line, the handset read as much as it could hold, and the XML it handed to its parser was cut off mid-document.

From the device's point of view the directory was malformed. From ours it was perfectly valid and had been sent in full. Both were true.

Why testing did not catch it

We test provisioning changes against physical handsets. That testing passed, and I do not think it was negligent — it was aimed at the wrong axis.

The failure needed two conditions at once: an affected model, and a directory long enough to exceed that model's buffer. Test devices on a test account have small directories. A modern handset with a generous buffer never truncates regardless. Either variable alone looks completely healthy, and testing tends to vary one thing at a time — a few devices with the same fixture data, or the same device across a few scenarios.

What was missing was a test where the size of the payload was the variable. Nothing in the change suggested payload size mattered, because in XML terms it does not.

The fix

One line, in the same helper:

fun cleanXml(xml: String) =
    xml.filter { it != '\n' && it != '\r' && it != '\t' }
       .trim()
       .replace("><", ">\n<")   // one tag per line

Strip every newline the builder produced, then deliberately put one back between each pair of tags. The result is still whitespace-normalised — no indentation, no stray tabs, which is what the original change was for — but now no single line is longer than one element. Line length stops scaling with directory size, and the buffer is never reached.

Newlines went from being noise to being structural. Nothing about the XML changed; the only thing that changed was where we chose to break the bytes.

What I took from it

"Valid" is a claim about a specification, not about a consumer. Our output conformed to the XML spec before and after. The spec was never the contract that mattered — the contract was with a particular firmware's reading strategy, and that contract was undocumented and, until it broke, unknown.

Formatting is part of the interface when your consumer is embedded hardware. On the web you can minify HTML, collapse JSON, strip newlines from anything, and the only cost is readability. Devices with kilobytes of RAM and a line-oriented read loop do not offer that guarantee. Treating layout as cosmetic is an assumption inherited from software that runs on generous machines.

Ask what varies with load, not just what varies with input. The change was correct for every document we tested and wrong for documents past a threshold nothing in our system knew about. Size-dependent failures hide well, because the small case is the one you naturally build a fixture for.

The change that caused this was two days old and looked like housekeeping. That is usually how it goes: the risky-looking changes get the scrutiny, and the whitespace cleanup goes in on a Monday afternoon.