How we fell and how we got up: calls incident 2026-10-23

Public incident report · CallPages Published because falling is inevitable; staying down is optional.

---

The thesis

Every technology company falls. The difference isn’t who avoids falling — that’s impossible — it’s what you do in the 24 hours after. This document exists because we believe how you react to a problem says more about you than the problem itself. We publish all of it: what broke, why, what we did, and what we changed so it doesn’t repeat. That is our reputation. We earn it this way, one fall at a time.

---

What broke

On October 23 we discovered our automatic voice provider rotation and failover wasn’t working. Calls followed a single route instead of spreading across providers with automatic failover. Worse: there was no record of which provider served each call, so the problem was invisible in our own data.

At the same time, the AI reports for some calls — transcription and summary — weren’t arriving. Calls that did happen were left with no detail in analytics. For a business owner, a call with no record is a call that never happened.

The day’s numbers: of 1,510 calls, 1,508 with no attributed provider. 41 interrupted, 17 partial, 15 with no report, 12 connection errors. The automatic failover that should protect every call had its counter at zero: it had never fired.

Why it happened (the real causes, no makeup)

1. The failover code was never wired to the real path. It existed, it was partly tested, but it wasn’t connected to the road calls actually travel. All traffic went through the old single-provider route.
2. Without attribution, the problem was invisible. If no record says which provider served each call, nobody can see that rotation isn’t happening. We were slow to see it because there was nothing to look at.
3. We had the signals and nobody watched them. Our systems did record the signs of capacity trouble. But there was no continuous pulse, no alerts, no one watching. The gap wasn’t data: it was attention.
4. The process allowed the fix to wait. A tested fix could sit for days without reaching production while production kept failing. Nothing prevented it.
5. An interrupted call lost its entire history. When the connection dropped mid-call, not even the partial was preserved. The business lost the whole call.

What we did in the hours after

• Real rotation and failover, wired to the call path. Every call now rotates across providers, fails over automatically, and records which provider served it. Verified with 49 automated tests.
• Total visibility. The usage panel shows every provider, its limits, and its status. What you can’t see you can’t fix; now everything is visible.
• System pulse. Every 5 minutes the health of calls, APIs, and pages is checked, with green/amber/red states and automatic alerts. Next time, we’ll see it in minutes, not the day after.
• New deploy process. No change reaches production if the automated checks are red. Red opens a priority ticket and blocks the deploy. It no longer depends on someone remembering.
• Public tickets for every failure mode. Every way a call can be lost has its ticket, its owner, and its tracking.

What we learned (deep learning)

Lesson 1: What’s not measured doesn’t exist. If a critical system property (which provider served each call) isn’t in the data, the system can be broken for weeks with nobody noticing. Now every call carries its attribution as a required field.

Lesson 2: Signals without watchers are decoration. Having metrics is useless if nobody looks at them. Observability isn’t a dashboard: it’s a pulse with thresholds, alerts, and someone (or something) obligated to react.

Lesson 3: The road to production is part of the system. A fix that doesn’t reach production isn’t a fix. Deploying has to be automated, verified, and blocked on failure — just like the code.

Lesson 4: Partial failure must leave a trace. A system that loses the full history when something interrupts is designed backwards. The partial is preserved, labeled, and shown. Never silently dropped.

Lesson 5: Publishing the incident is the final fix. Writing this forced us to understand it better. Publishing it forces us to honor it. Reputation isn’t declared: it’s demonstrated, incident by incident.

Our commitment (the DNA)

We will fall more times. It’s the price of building fast. What we promise is this:

1. Every priority incident leaves a public report like this one. No exceptions, no sugarcoating.
2. Every report ends in structural changes, not patches. If we only fixed the symptom, we learned nothing.
3. We measure what matters and we watch it always. The pulse never turns off.
4. No deploy ships on red. The process protects us from ourselves.

Falling isn’t failure. Staying down is. And hiding the fall is worse than the fall itself.

---

CallPages — pages with AI voice agents. We fall, we get up, we publish it.