Writing
Notes on problems worth writing up — systems I've built, things that broke in interesting ways, and decisions I would make differently a second time. 25 pieces, grouped by the system they came out of.
Launching a new brand4 posts
-
What you cut when the date is fixed
Joining a launch two weeks out with the date immovable. Most of the work was deciding what not to build — and noticing which of the existing cuts were wrong.
-
Letting the payment provider own the money
Stripe holds the pricing, the subscription and the cancellation; our service listens. That hands pricing decisions back to the people who make them, and leaves exactly one new problem.
-
An order that can be run again
One button buys a phone number. Behind it is a sequence of steps against systems that fail occasionally, and the worst failure is the one that produces no error at all.
-
Support needs it on day one
Nobody had specified what the support team would need. Low traffic is not the same as low stakes: one customer nobody can help is still a bad launch day.
The platform migration8 posts
-
The hard part was the devices already out there
Rewriting a provisioning server from Python to Kotlin, and cutting thousands of desk phones over to it by DNS. The rewrite was the easy half.
-
The whitespace was load-bearing
Removing newlines from a valid XML document took out desk phone directories across an entire handset range. The XML stayed valid; the line length did not.
-
Nonces don't fit in one process
Digest authentication worked fine on a single Python process. Running several instances behind a load balancer turned a local variable into a distributed systems problem.
-
Six vendors, one API
You can abstract the transport. You cannot abstract the semantics. Notes on writing one provisioning interface across six manufacturers' firmware.
-
Cheaper for the server, dearer for the phone
Splitting a monolith stopped any one service holding a whole directory in memory. It also made large directories slow to assemble — and the slowest client sets the limit.
-
Everyone checks in at once
Give every phone the same check-in interval and they synchronise. The obvious fix — random jitter — would have rebooted the estate on every cycle.
-
Writing to both
You cannot move a live platform to a new database in one step. Dual writes are how you make the migration boring — and most of the work is in the failure handling.
-
Expected failures are not errors
Rolling tracing and structured logging across a platform. The instrumentation was the easy half; making the output mean something took longer.
Regulated telecoms3 posts
-
The address that has to be right
Two overlapping models had grown up for the address that determines where a 999 call is routed. Consolidating them meant deciding which one had ever been true.
-
Porting a number
Moving a phone number between providers is a regulated, document-backed handover between competitors, and the failure mode is a business losing the number on its van.
-
Someone else's phone bill
Toll fraud costs real money by the minute. The useful controls turned out to be graduated ones — risk scores that trigger verification rather than walls that block.
Mobile clients4 posts
-
The socket is not the call
Building real-time calling into a mobile app means the operating system, your app and a websocket all hold an opinion about whether a call is happening. They disagree.
-
Presence is not a boolean
The little light showing whether a colleague is free looks like one bit. It is a composite of registration, call state, do-not-disturb and queue availability, delivered to every client.
-
The rewrite that was never a project
Building a chat client inside a phone app over three years, and moving it from UIKit to SwiftUI in a single commit nobody scheduled.
-
The store is part of the system
Shipping one codebase as several branded apps through two app stores. Release is not a deployment step, it is a dependency with its own deadlines and its own opinions.
Platform and internal tools6 posts
-
The tool your colleagues live in
Maintaining the internal portal a company runs on. Your users sit near you, nothing is ever prioritised until it breaks, and it is the union of every other system you own.
-
One person, several accounts
Single sign-on assumes an identity maps to an account. In a white-label platform it often maps to four, and the login flow has to cope before the user has proved anything.
-
Hiding a button is not a permission
Adding granular permissions to a portal that grew up without them, and keeping the interface honest about what a user can actually do.
-
Reports are a scheduling problem
Building call reporting end to end. The queries are the easy part; the hard parts are recurrence, drilldowns and deciding what a number means.
-
A catalogue is a query
Selling hardware, numbers and contract addons through partners. What you can order depends on who you are, what you already have, and how long you signed up for.
-
Decommissioning is the milestone
Replacing the helpdesk your support team works in, without them stopping. The migration is the easy half; turning the old one off is the part that counts.