Writing

Notes on problems worth writing up — systems I've built, things that broke in interesting ways, and decisions I would make differently a second time. 25 pieces, grouped by the system they came out of.

Launching a new brand4 posts

  • What you cut when the date is fixed

    Joining a launch two weeks out with the date immovable. Most of the work was deciding what not to build — and noticing which of the existing cuts were wrong.

  • Letting the payment provider own the money

    Stripe holds the pricing, the subscription and the cancellation; our service listens. That hands pricing decisions back to the people who make them, and leaves exactly one new problem.

  • An order that can be run again

    One button buys a phone number. Behind it is a sequence of steps against systems that fail occasionally, and the worst failure is the one that produces no error at all.

  • Support needs it on day one

    Nobody had specified what the support team would need. Low traffic is not the same as low stakes: one customer nobody can help is still a bad launch day.

The platform migration8 posts

  • The hard part was the devices already out there

    Rewriting a provisioning server from Python to Kotlin, and cutting thousands of desk phones over to it by DNS. The rewrite was the easy half.

  • The whitespace was load-bearing

    Removing newlines from a valid XML document took out desk phone directories across an entire handset range. The XML stayed valid; the line length did not.

  • Nonces don't fit in one process

    Digest authentication worked fine on a single Python process. Running several instances behind a load balancer turned a local variable into a distributed systems problem.

  • Six vendors, one API

    You can abstract the transport. You cannot abstract the semantics. Notes on writing one provisioning interface across six manufacturers' firmware.

  • Cheaper for the server, dearer for the phone

    Splitting a monolith stopped any one service holding a whole directory in memory. It also made large directories slow to assemble — and the slowest client sets the limit.

  • Everyone checks in at once

    Give every phone the same check-in interval and they synchronise. The obvious fix — random jitter — would have rebooted the estate on every cycle.

  • Writing to both

    You cannot move a live platform to a new database in one step. Dual writes are how you make the migration boring — and most of the work is in the failure handling.

  • Expected failures are not errors

    Rolling tracing and structured logging across a platform. The instrumentation was the easy half; making the output mean something took longer.

Regulated telecoms3 posts

  • The address that has to be right

    Two overlapping models had grown up for the address that determines where a 999 call is routed. Consolidating them meant deciding which one had ever been true.

  • Porting a number

    Moving a phone number between providers is a regulated, document-backed handover between competitors, and the failure mode is a business losing the number on its van.

  • Someone else's phone bill

    Toll fraud costs real money by the minute. The useful controls turned out to be graduated ones — risk scores that trigger verification rather than walls that block.

Mobile clients4 posts

  • The socket is not the call

    Building real-time calling into a mobile app means the operating system, your app and a websocket all hold an opinion about whether a call is happening. They disagree.

  • Presence is not a boolean

    The little light showing whether a colleague is free looks like one bit. It is a composite of registration, call state, do-not-disturb and queue availability, delivered to every client.

  • The rewrite that was never a project

    Building a chat client inside a phone app over three years, and moving it from UIKit to SwiftUI in a single commit nobody scheduled.

  • The store is part of the system

    Shipping one codebase as several branded apps through two app stores. Release is not a deployment step, it is a dependency with its own deadlines and its own opinions.

Platform and internal tools6 posts

  • The tool your colleagues live in

    Maintaining the internal portal a company runs on. Your users sit near you, nothing is ever prioritised until it breaks, and it is the union of every other system you own.

  • One person, several accounts

    Single sign-on assumes an identity maps to an account. In a white-label platform it often maps to four, and the login flow has to cope before the user has proved anything.

  • Hiding a button is not a permission

    Adding granular permissions to a portal that grew up without them, and keeping the interface honest about what a user can actually do.

  • Reports are a scheduling problem

    Building call reporting end to end. The queries are the easy part; the hard parts are recurrence, drilldowns and deciding what a number means.

  • A catalogue is a query

    Selling hardware, numbers and contract addons through partners. What you can order depends on who you are, what you already have, and how long you signed up for.

  • Decommissioning is the milestone

    Replacing the helpdesk your support team works in, without them stopping. The migration is the easy half; turning the old one off is the part that counts.