Retrospective: how broken pages and broken infrastructure reached production — and why staging did not stop them

On 24 October 2026 our own checks confirmed what the owner found first: pages failing on the paths owners actually take, and an infrastructure layer where one account’s dashboard showed another account’s Cloudflare connection for eleven days. This retrospective names what broke, why staging could not catch it, and the gates that replace discipline with mechanism.

This was not a coding accident and it was not bad luck. Production was protected by discipline, memory, and good intentions — not by gates. When a check depended on a person remembering to perform it, on staging data that resembled production, or on a monitor that could fail silently, it eventually failed in exactly that way. The owner found the defects by using the product. By definition, that means the system had already failed.

Date: 24 October 2026

Scope: CallPages web pages and editor surfaces; account infrastructure — Cloudflare connections, Gemini keys, and shared provider pools

Trigger: Owner report, 24 October 2026 — the main account dashboard showed another account’s Cloudflare connection, and the main account was reported as having no usable Gemini key

Classification: P0 process failure. Customer trust, account isolation, and release integrity were all at risk, whether or not every defect caused a visible outage for every user.

Status of the underlying defects: Confirmed in production data on 24 October 2026. Correction is a separate work stream; this document does not claim the defects are fixed.

1. What shipped broken: the pages

The page failures share a pattern that is more damning than any single bug: the system tested that things existed, and reported success, while the action the owner actually needed either failed or was never exercised at all.

Presence was tested. Function was not.

The clearest documented case is the Android save failure of 17 October 2026. Write endpoints declared an untyped request shape that the real client library rejects at the moment of invocation. Every save through those methods threw on the owner’s device — while the unit suite reported 155 of 155 tests passing, because no test ever instantiated the real client interface. A single contract test would have failed the build on the day that shape was written. This is the canonical example for the whole retrospective: a green suite that cannot see the failure it claims to prevent is worse than no suite, because it manufactures confidence.

The editor failed on the path the owner actually takes

The page-editor outage, also in October, had the same shape on the web. The editor loaded inside a hidden panel; following the Edit button, or opening a direct editor link, did not activate the panel that contained it. The editor appeared only after an unrelated back-and-forward navigation happened to run the tab-switching code. Production and staging served identical broken code. Three navigation paths failed in reproduction; after the fix, twelve of twelve scenarios passed — which proves the scenarios were testable all along. They simply had not been required before release.

Calls, analytics, and the store editor repeated the lesson

• New-page calls (13 October): calls opened from one host form died while the same pages connected from the other, because the two hosts are separate browser origins with separate microphone-permission stores. Page configurations passed 687 of 687 readiness checks. The configuration was innocent; the user’s real entry path was not tested.
• Analytics (23 October): a ticket calling for sophisticated analytics was closed in a state where the event fan-out was dead because a required key was absent. Acceptance — real events visible in the analytics console — had not been observed. The closure claim ran ahead of the evidence.
• Store and collection editor: at the time of this retrospective the owner’s priority surface still carries reported defects in basic product management — deletion, visibility control, and mobile editing quality. Basic create-read-update-delete actions on the primary commercial surface are not a polish item; they are the product.

Finding P-1. Page releases were verified at the level of code merged, suite green, page renders. They were not consistently verified at the level the owner experiences them: open the real link, take the real path, complete the real action, read the result back. Every defect in this section lives in that gap.

2. What shipped broken: the infrastructure

On 24 October 2026 the owner opened the dashboard for the main account, vasilyespana@gmail.com, and found it showing a Cloudflare connection belonging to a different account, vasilypfknr@gmail.com. Production data confirmed the report. This was not a display error.

Invariant that must always hold	State confirmed in production, 24 Oct 2026
Each Cloudflare connection belongs to the account whose owner email matches it	BROKEN. The only connection row was stored under vasilyespana@gmail.com but pointed to the vasilypfknr@gmail.com Cloudflare account. It had been in that state since 13 October 2026 — eleven days in production, undetected by any system.
The main account uses its own paid Cloudflare account	BROKEN. The paid account belongs to vasilyespana@gmail.com, but it is not the account linked in that row.
Every account has a working Gemini key of its own	BROKEN. The main account has a Gemini row, but its status was degraded at its last check (24 October, 00:31), so the system treats it as unusable and falls back to a central key. vasilyukreino@gmail.com and vasilypfknr@gmail.com had no Gemini rows at all.
Shared pools add real, shared capacity on top of each account’s own key	BROKEN IN SUBSTANCE. The shared pool had no donor Gemini rows in the shared configuration — only the central row. A pool with no members is a label, not capacity.
Keys that exist are connected where they are used	PARTLY TRUE, AND THAT IS THE POINT. The owner supplied evidence of five distinct Gemini keys in five separate Google projects. The keys existed on the provider side. They were not registered, probed, and visible per owner inside CallPages. Existence upstream was mistaken, in effect, for connection downstream.

Why this class of defect is the most serious in this document

A broken button is visible. A broken identity binding is silent and corrosive. When an account can display, and potentially act through, another account’s infrastructure connection, three things are damaged at once: the owner’s trust that the dashboard tells the truth, the isolation guarantee every other account depends on, and the meaning of every health indicator built on top of those bindings. If the binding layer is wrong, a green dashboard can be lying politely.

No Gemini key values are reproduced in this retrospective, and none should ever appear in a report, a screenshot, a ticket, or a chat. Key-to-account assignment must likewise never be guessed: an assignment inferred from a screenshot would risk repeating, in the provider layer, the exact cross-account error this document exists to correct. Each key is registered inside its owner’s account, or its mapping is confirmed by its owner — there is no third route.

Finding I-1. No automated invariant ever asked the elementary question: does every connection, key, and pool membership in this account belong to this account, and does it work right now? The question was never encoded, so it was never asked — for eleven days.

3. Timeline: the warnings were there

When	What happened — and what it was telling us
9–17 Oct 2026	A build monitor polled with its error output swallowed. When its authentication died, every poll failed, it reported a success-shaped result for eight days, and its saved state froze while staging advanced dozens of commits past it. Lesson available: a monitor that cannot scream is an outage multiplier.
13 Oct 2026	New-page calls failed on one host form while succeeding on the other; the diagnosis was proven from production evidence, including the owner’s own phone. The same day, the mismatched Cloudflare connection that the owner would find on 24 October was created. A tenant credential exposed in a screenshot that day also remained unrotated past the same-day rule.
17 Oct 2026	The save-that-could-never-succeed failure and the page-editor outage were both live in production. A formal gap audit scored quality enforcement at 0 fully in place, 10 partial, 2 missing — and recorded that the production deploy path ran with zero test steps and that continuous integration ran no test suites on pull requests.
17 Oct 2026	Of the last 30 automation runs audited, 30 had failed — deploys, release notes, analytics posting, device gates, and end-to-end runs — while deploys still landed. Red had been fully normalized. A signal that is always red carries no information.
23 Oct 2026	The analytics fan-out gap was named honestly: capture restored, real events not yet observed — a partially closed ticket with the gap declared. This was the standard working as intended, and it was the exception, not the norm.
24 Oct 2026, 00:31	The main account’s Gemini key was checked and left in degraded status. No escalation converted a degraded sole key into an incident.
24 Oct 2026	The owner, using his own dashboard, found the crossed Cloudflare connection and the missing-key state, and reported both. Detection credit belongs to the owner. That is the failure this retrospective is about.

Reading the timeline correctly. There is no single bad deploy to blame and no single engineer to correct. There is an eleven-day window in which a P0 identity defect sat in production, a monitor-blindness incident, a normalized wall of red automation, and an audit that had already named the missing gates in writing. The system produced the information. It did not produce the obligation to act on it. That distinction — information versus obligation — is the heart of the root-cause analysis that follows.

4. Why staging did not catch any of it

Staging is not a smaller production. In this system, at the time of these failures, staging differed from production in the four dimensions that mattered most — and the release path asked staging questions it was structurally unable to answer.

4.1 Staging data did not contain the defect

The Cloudflare mismatch was a data defect: one stored row, owned by one account, pointing at another account’s provider account. Code tests do not find that; only an invariant check run against the real bindings does. Staging fixtures held no multi-account, cross-checked set of owner-to-provider bindings, so even a perfect staging run had nothing to catch. The same applies to missing Gemini rows and an empty shared pool: absence is invisible to a test that never enumerates what must be present, per account.

4.2 Staging identity and secrets diverged from production

Staging authentication and provider secrets had a documented history of diverging — including staging preview secrets being wiped by rebuilds more than once, and a period when staging reported a key as missing while production reported it set. A staging environment whose logins and keys are not production-like cannot validate login, provider probing, or fallback behaviour, and teams learn to wave its failures through — which then waves production failures through by habit.

4.3 The deploy path did not require tests to pass

The gap audit verified that continuous integration ran no test suites on pull requests, and that the production deploy workflow contained zero test steps. Branch protection that could have required checks was unavailable on the repository’s plan, and no compensating mechanism with the same blocking force replaced it. In practical terms: a red suite blocked nothing, a missing test blocked nothing, and merge plus deploy was treated as evidence of readiness.

4.4 Tests measured presence, mocks, and happy paths

Interface tests ran against a fake backend; parity checks established that sections exist; page-configuration checks established that data was well formed. None of those can fail when the real client throws at invocation, when the real entry host breaks a call, or when the real account binding points at the wrong tenant. The test estate was large — over a thousand web tests — and still blind in the specific places production broke. Volume is not coverage.

4.5 Nobody was obliged to look, and looking was hard

Monitors failed silently, error streams from entire editing surfaces produced no records, a crash-reporting path shipped without ever producing a row, and chronic automation failure had no owner and no age limit. Detection therefore defaulted to the only sensor left in the loop: the owner, using the product, in production. That is the most expensive sensor a company can run — paid for in trust.

5. Root causes: five whys, one system

Why did broken work reach production?

1. Why were the defects not caught before release? Because no automated check exercised the failing condition: the real client invocation, the real entry path, the real per-account bindings.
2. Why did no such check exist? Because completion was defined as code written, suite green, and pull request merged — not as a user action verified on the real surface with production-like data.
3. Why was completion defined that way? Because the quality standard existed as prose and discipline, not as blocking mechanisms in the pipeline. At audit, zero of its twelve rules scored as fully enforced.
4. Why did mechanisms not replace discipline? Because each missing gate was individually tolerable and collectively invisible: unavailable branch protection, a deploy workflow without test steps, monitors without heartbeats, and chronically red automation were each treated as background conditions rather than as defects with owners and deadlines.
5. Why were those conditions tolerated? Because no metric made them hurt: no error budget, no owner-found-defect count per release, no failure-age limit for red automation, no scorecard naming what was not verified. What is not measured as a loss feels free — until the owner pays.

The five systemic root causes

Root cause	How it produced both failure classes
RC-1. Mechanisms were replaced by intentions	Test-first, verify-on-the-real-surface, and rotate-exposed-secrets rules all existed in writing and all failed in practice, because nothing in the pipeline could stop a release that skipped them.
RC-2. No identity invariant layer	Nothing continuously asserted that accounts, provider connections, keys, and pools belong to their owners and work. Cross-account defects had an unlimited lifetime — eleven days, ended only by a human glance.
RC-3. Staging was not a production rehearsal	Different data, different secrets, no multi-account binding fixtures, no production-parity checklist. Staging passing meant staging passed, and was silently read as production passing.
RC-4. Signals without owners or consequences	Thirty of thirty automation runs failing changed nothing. A degraded sole Gemini key changed nothing. Signals that cannot block, page, or file work are decoration.
RC-5. Claims ran ahead of evidence	Tickets and releases were called done at the level of code state, not live-surface proof — the analytics closure is the documented example. Every unsupported green teaches the system to produce more.

What this does not conclude. Not that the team lacked a standard, effort, or test volume — all three existed in abundance. Abundance without enforcement is how a sophisticated operation ships elementary defects: effort flows to what is visible, while the gates that catch a crossed wire stay perpetually next in line.

6. Impact assessment

Dimension	Assessment
Account isolation and trust	Severe. The main account’s dashboard presented another account’s infrastructure as its own for eleven days. Even if no data changed hands, the guarantee the product sells — your account, your infrastructure — was demonstrably false on its most visible surface.
Service continuity	Material. The main account’s voice and AI features were running on fallback capacity because its own key was degraded, while two accounts had no key of their own and the shared pool contributed no donor capacity. The system was one central-key failure away from a wider outage, and nothing was counting down to that event.
Owner time and confidence	Severe. The owner acted as integration tester, monitoring system, and incident reporter for defects the pipeline existed to catch. Each such incident spends confidence that feature work cannot buy back.
Release credibility	Severe. If staging and production verification missed an identity mismatch visible on the dashboard’s front page, no other release claim from the same process can be taken at face value until the gates change. Credibility must be re-earned release by release, with evidence.
Data integrity	Bounded, with a warning. The confirmed defect was a mis-pointed connection row, correctable directly — not a case for restoring an old database state, which would destroy days of legitimate work. The warning is that no inventory existed to prove this was the only crossed binding. That inventory is now a required action, not an assumption.

The decision rule this establishes

From this retrospective forward, an owner-found P0 or P1 defect is classified automatically as a process incident in addition to a product defect. Fixing the instance closes the product ticket. It does not close the incident. The incident closes only when the gate that would have caught the class exists, is demonstrated failing on the old state and passing on the new one, and is wired to block a release. This rule already existed in the quality standard as its meta-rule; this retrospective promotes it from principle to release condition.

7. Action plan I: the stop-line (first 72 hours)

These actions come before any new feature release. Owners are roles; the engineering lead owns the plan as a whole and reports its state daily until every item in this section is closed with evidence.

#	Action	Owner	Acceptance evidence — the action is not done without it
A1	Freeze non-essential releases. Only defect corrections and gate work ship until A2–A5 pass. The daily release train resumes only on a green identity and smoke report.	Engineering lead	Freeze stated in the release record; any exception named, with reason and expiry, in the same record.
A2	Full binding inventory. Enumerate every account, every Cloudflare connection, every provider key row, and every shared-pool membership in production. Assert, per row: stored owner equals provider-account owner.	Infrastructure owner	A complete inventory table with a pass or fail per row — not a sample. Every fail becomes a P0 ticket the same day.
A3	Correct the confirmed mismatch directly. Re-point the main account to its own paid Cloudflare account and restore the crossed row to its true owner. No bulk restore to an earlier database state.	Infrastructure owner	Read-back in each affected owner’s dashboard showing that owner’s own connection — verified on the live surface, per owner.
A4	Every account gets a working key of its own. Re-probe the main account’s degraded Gemini key; register a key inside each account that has none, through that account’s own infrastructure surface, with the mapping confirmed by the owner or by registration while signed in as that owner. Never assign a key by inference.	Infrastructure owner	Per account: key present, probe passes, status healthy in that owner’s dashboard. A screenshot with a key value in it is a security incident, not evidence.
A5	Make the shared pool real. Populate the pool with confirmed donor keys and prove that pooled capacity is actually selected when an account’s own key is unavailable.	Infrastructure owner	A controlled test: disable one test account’s own key, observe a successful call served from the pool, restore. Logged with provider attribution.
A6	Rotate any credential exposed during this incident. Any key value that appeared in chat, screenshots, or documents during diagnosis is treated as exposed and rotated the same day.	Security owner	Rotation confirmed per credential; the exposed value no longer probes successfully.

8. Action plan II: the gates that make staging mean something

#	Gate	By when	How it blocks, and how we know it works
G1	Identity invariant gate. An automated suite, run against staging fixtures and against production in read-only mode after every release, asserting for every account: connection owner equals account owner; a healthy own key exists or an explicit exception is recorded; pool membership is non-empty where a pool is claimed.	Week 1	A mismatch fails the release pipeline and files a P0 automatically. Proven by a seeded-mismatch test: the gate must go red on a deliberately crossed fixture before it is trusted green.
G2	Tests in the deploy path. The full web and mobile unit suites, plus contract tests, run on every pull request and again as a required step before any production deploy. A red suite stops the deploy — in the workflow, not by convention.	Week 1	Workflow definitions show the test steps gating the deploy job. A deliberately failing test on a throwaway branch must demonstrably prevent deploy.
G3	Real-client contract coverage. Every client write method is exercised by a test that builds the real client against the real API shape. Untyped wildcard request bodies are rejected by a static check.	Weeks 1–2	Coverage report: 100% of write methods. The October save defect, reintroduced in a test branch, must fail this gate.
G4	Production-like staging. Staging carries seeded multi-account fixtures mirroring production’s binding shapes, its own stable secrets managed as code, and a parity checklist comparing staging and production configuration before each release train.	Week 2	Fixture inventory plus a signed parity checklist per release. Any staging-versus-production configuration drift is a listed, owned item — never a surprise.
G5	Real-surface smoke journeys. After every release, automated journeys on the production surface itself: sign in as each test owner, open the dashboard, verify the owner’s own infrastructure is shown, open a page by its real public link on each host form, place a test call, and complete one create-edit-delete cycle in the store editor.	Week 2	Journey results attached to the release scorecard with the live build identifier. A release without a passing smoke record is reported as unverified, in those words.
G6	Function-level parity. Parity checks assert that core actions complete end to end — create, edit, save, delete, hide — on each surface that claims parity, replacing presence-only checks.	Weeks 2–3	Parity percentage computed from these checks per release; a parity claim in release notes must cite the measured number.

9. Action plan III: signals, claims, and governance

#	Action	By when	Acceptance evidence
S1	Every monitor gets a heartbeat and a loud failure. No monitor may report a success-shaped result when its own polling, authentication, or upstream has failed. Failure files work automatically.	Week 2	Monitor inventory at 100%: name, cadence, failure signal, heartbeat, owner. A game-day drill kills one monitor’s credentials and observes the alarm.
S2	End normalized red. Every chronically failing automation run gets a named owner and a fix-or-delete decision. No failure may remain red and unowned for more than five working days.	Week 2, then continuous	Failure-age report in the weekly scorecard; the count of unowned red runs is zero and stays zero.
S3	Provider health becomes an incident trigger. A sole account key entering degraded status, a provider circuit opening, or pooled capacity reaching zero files a ticket and alerts immediately — it does not wait for a dashboard visit.	Week 1	Drill: degrade a test key and observe the ticket and alert, with timestamps.
S4	Errors become bugs automatically. Every error cluster from every surface — editing surfaces included — files or increments a deduplicated bug. Silent failure paths are defects of the same severity as the failures they hide.	Week 3	Daily digest exists and is read: new clusters, top clusters, owner-found defects. A sampled audit shows user-visible failures matching telemetry records.
S5	Evidence-based closure language. Tickets and releases may be called done only with live-surface evidence attached. Partial states are labelled partial, with the remaining gap named — as the analytics ticket eventually was, by rule rather than by exception.	Immediate	Pull-request and ticket templates carry the evidence checklist; closures without it are reopened on audit.
S6	Release scorecard per train. Every release publishes: build identifier, gates passed, smoke journeys passed, identity invariant result, rollback point, and an explicit list of what was not verified.	Week 1, then every release	Scorecards archived with the release. The not-verified list is expected to shrink to empty for P0 surfaces — and to say so honestly until it does.

Governance

Weekly, for four weeks, then monthly: review this plan’s open items, the owner-found-defect count, gate failure-age, and error-budget state. The review closes items on evidence, not on elapsed time. Exceptions expire. Any gate may be excepted for one release only with a written reason and an expiry recorded in the release scorecard. An expired exception blocks the next release.

Repeat causes escalate. A defect class that recurs after its gate exists raises the severity of both the defect and the gate failure, and the gate — not only the instance — is fixed.

10. Public by default: this retrospective is not an internal note

Firm requirement, applying to this retrospective and to every retrospective that follows it. A retrospective is a public document. It is published openly, for public visibility into how we fail, how we adapt, how we respond to a failure, and how we strengthen and harden our processes as a result. Hiding it internally would defeat its purpose.

Publication order is part of the requirement

1. First on our own website. This retrospective is published first on callpages.me, in full, as the canonical public version. That version is the one that is linked, quoted, and kept current if the action plan’s status changes.
2. Then shared on social media. Social posts follow the website publication and link back to it. A retrospective is never published social-only: a social post without the full document on our own site does not satisfy this requirement.
3. Never filed away as a private lesson. Internal tickets, gates, and owners are how the work gets done. They are not a substitute for the public account of what happened and what changed.

Why publish a failure

Customers do not trust companies that claim they never fail. They trust companies whose failures they can inspect — where the account of what went wrong is specific, the response is verifiable, and the hardening that followed can be checked against the next release. Published plainly, a retrospective is therefore not a confession to be minimized. It is evidence of how seriously the operation takes its own defects, and it is how learnings become reputation: a reader who watches a company name its failure, fix its gates, and report the result has a reason to believe the next claim it makes.

That framing governs the tone as well as the venue. This document names real defects, real durations, and real missing gates, without softening them into generic lessons learned — because a vague public retrospective builds no trust at all. Equally, it publishes no credentials, no customer data, and no private information: transparency is about our process and our failures, never about exposing anyone else.

Acceptance for this section. Publication is complete only when the full retrospective is live on callpages.me and a social share linking to that page exists. Until both are true, the publication status of this retrospective must be reported as not yet published — in those words — in any status update that mentions it.

11. How success will be measured

The plan is not complete when its tasks are checked off. It is complete when the numbers that describe this failure move, and stay moved, for two consecutive release cycles.

Measure	At the time of the failure	Target
Owner-found P0/P1 defects per release	At least two in a single report, 24 October — infrastructure identity and key state	Zero, sustained. This is the headline measure; everything else explains it.
Identity invariant violations in production	One confirmed, undetected for 11 days	Zero, with detection time under one release cycle for any seeded or real violation
Accounts with a healthy key of their own	Main account degraded; two accounts with no key row; pool with no donors	100% of accounts, reported per account in the health view
Production deploys preceded by passing test and contract gates	Deploy path contained zero test steps	100% of deploys
Client write methods with real-client contract coverage	Absent — the canonical save failure was invisible to the suite	100% of write methods
Automation runs red and unowned for over five days	30 of 30 recent runs failing, unowned	Zero
Monitors with heartbeat and distinct failure signal	Partial; one monitor blind for eight days while reporting success	100% of the inventory
Releases with a complete scorecard, including a not-verified list	Not a standard artifact	100% of releases

Closing statement. The owner asked a fair and uncomfortable question: how did staging pass what production could not survive? The honest answer is that staging was never given the data, the identity, the gates, or the authority to stop these releases — and production verification did not perform the checks either, so the first full test of the system was the owner using it.

The commitment that follows from this retrospective is not to try harder inside the same process. It is to change what the process is capable of certifying: bindings proven per account, actions proven on the real surface, tests that can actually fail, monitors that can actually scream, and a release that cannot call itself verified until all of that is true. Red becomes green when the gates say so — on evidence the owner can inspect — and not one release earlier.

Sources: owner report and confirmed production check, 24 October 2026; Quality Standard and Gap Audit, 17 October 2026 (remediation R1–R16, carried into Sections 7–9); incident records of 13, 17, and 23 October 2026. No credential values are included.

How it works

1. The corrective standard: no release is verified until an automated check, running against production-like identity and data, proves the real user action works on the real surface — and a failing check blocks the release.
2. The stop-line comes first: freeze non-essential releases; complete a full binding inventory of every account, Cloudflare connection, provider key and shared-pool membership; correct the confirmed mismatch directly; give every account a working key of its own; make the shared pool real; rotate any credential exposed during the incident.
3. Then the gates: an identity invariant gate, tests inside the deploy path, real-client contract coverage, production-like staging, real-surface smoke journeys after every release, and function-level parity checks.
4. And the governance: every monitor gets a heartbeat and a loud failure; chronically red automation gets a named owner or is deleted; provider health files incidents automatically; errors become bugs automatically; and closure language requires live-surface evidence.

Related reading

• Release notes 2026.10.24

See how CallPages works →

---

Innovation by Vasily Starostenko, founder of CallPages.