Expected failures are not errors

Adding observability to a platform sounds like a procurement decision followed by some configuration. Pick a backend, add the agent, restart everything, look at the dashboard.

The instrumentation genuinely is close to that. What took the time was making the output mean anything — and the single biggest contributor was that a large share of what our services logged as errors were not errors at all.

The provisioning server taught me this

A provisioning server sits on the public internet and answers requests from devices. A meaningful fraction of what reaches it is never going to succeed: a handset with a MAC address that is not on any account, a device pointed at us by a customer who has not bought anything, a scanner, a misconfigured phone retrying every thirty seconds forever.

Every one of those is a request that cannot be fulfilled. It is not a fault. Nothing is broken, no code path misbehaved, and there is no action anyone would take. But the obvious implementation treats "could not fulfil this request" as an exception, logs it at error level, and attaches a stack trace.

Do that at internet scale and your error log is almost entirely noise. Which means nobody reads it. Which means the one genuine error in ten thousand lines goes past unseen, and you have spent money on observability to obtain a slightly more expensive way of not noticing things.

So a real chunk of the work was severity triage: reducing expected conditions to warnings or information, and — separately, and just as valuable — stopping expected failures from producing stack traces at all. A stack trace is a claim that something unanticipated happened in a place worth looking at. Attaching one to a routine rejection is a lie your log tells you several thousand times a day.

Traces that do not join up are worse than none

The tracing rollout went in two stages: every service, and then the gateway in front of them.

The technically awkward part was context propagation. A request enters through the gateway, hits one service, which calls two more over gRPC, one of which publishes a message another consumes later. For that to be one trace rather than five, the context has to survive every one of those boundaries — including the ones where the framework does not do it for you, and the ones where a thread or a coroutine changes underneath you.

Getting this half-right is genuinely worse than not doing it. A trace that stops at a service boundary tells you the request vanished, which is a fact about your instrumentation being presented as a fact about your system. People then reason about the fake gap. Until propagation works end to end, the traces are actively misleading rather than merely incomplete.

Telemetry must not take the service down

Two decisions I would repeat.

The trace destination is optional. If the collector is unreachable, misconfigured, or having a bad day, services carry on. That sounds obvious and is easy to get wrong, because the natural implementation fails startup on a bad endpoint — and then an observability problem becomes an availability problem, which is exactly backwards. Telemetry is how you watch the system; it must never be something the system depends on.

You can ship to two destinations at once. Adding an optional secondary destination meant we could evaluate a different backend, or move between them, without a flag day. Observability tooling needs migrating like anything else, and the ability to dual-ship is what makes that boring.

There is a pleasing symmetry there with dual-writing data during a database migration. Same idea, different layer: write to both, compare, switch reads, then stop.

Instrument before you need it, not after

The thing I would emphasise most is about timing rather than technique.

When we cut a provisioning server over by DNS, monitoring and alerting were configured in advance — before go-live, not after. That was deliberate, and it is the single cheapest decision in the whole exercise.

You cannot assess a cutover you cannot see. Without instrumentation in place beforehand, the question "is this working?" gets answered by waiting to see whether anybody complains, which is both slow and a poor test — most of what goes wrong in a migration is degraded rather than broken, and nobody phones up about slightly wrong.

Adding observability after an incident is the normal pattern and it is always one incident too late.

The domain-specific signals are the useful ones

Generic telemetry tells you about latency, throughput and error rates. Useful, and not what anyone actually asks.

The signals that earned their place were specific: logging requests from unrecognised hardware identifiers, so you can tell a customer why their handset will not provision. Flagging number ports that have sat with a carrier too long. Alerting when a call arrives with no source address — something that should be impossible and therefore wants a human to look.

Those are not observability in the dashboard sense. They are the system telling you about states that are meaningful in the business, and they get used because the person reading them can act.

What I took from it

Severity is a product decision. The question is not "did something fail" but "would anybody act on this". Anything nobody would act on is not an error, whatever the code thinks, and log levels that do not reflect that make the whole output worthless.

Half-propagated context is a lie. Incomplete traces do not degrade gracefully — they invent gaps that people then investigate. Finish propagation before trusting anything the traces say.

Never let telemetry become a dependency. Optional destinations, non-fatal failures, and the ability to ship to two places at once. The observability stack should be able to fall over without anyone outside the team noticing.

Instrument before the event. Monitoring added before a migration is worth several times the same monitoring added after it, and the cost is identical.