The Safest Way to Replace a System Everyone Relies On

The safest way to replace an internal system everyone relies on is to ship the replacement read-only first. A read-only app is useful on day one — it can be a faster, clearer view of the same data — and it is structurally incapable of corrupting the system it will eventually replace. Where it disagrees with the incumbent, the disagreement surfaces as a question to investigate, not as damage to undo.

The alternative has a name — the big-bang cutover — and a well-documented worst case. In April 2018, TSB Bank moved roughly five million customer accounts to a new platform in a single weekend. The independent investigation that followed found the platform was not ready, and the regulatory and compensation bills ran into the tens of millions of pounds. This post treats TSB as one dated, thoroughly investigated example of a failure mode — not as evidence of how often migrations fail. No source in our research supports an industry-wide failure rate, so we make no claim about one.

What follows is a rollout pattern: why the established migration literature already points this way, how to enforce read-only at the database rather than trusting the application, a graduation ladder each surface must climb before it earns write access, and the counter-intuitive case for surfaces that should stay read-only forever.

On the evening of Sunday 22 April 2018, TSB migrated roughly five million customer accounts in one weekend from Lloyds Banking Group's platform to Proteo4UK, a new platform built by Sabis, the IT arm of TSB's parent Sabadell. In the days that followed, Computer Weekly's reporting on the independent investigation records that customers were locked out of accounts, some saw money missing, and some could see other customers' accounts and transactions. Internet and mobile banking were, per the report, “unstable and almost unusable.”

The investigation — commissioned from law firm Slaughter and May, a 262-page report that Computer Weekly says is reported to have cost around £25 million — concluded: “The platform was not ready to support TSB's full customer base and Sabis was not ready to operate the platform.” Its finding on the method is the part every internal tools team should read twice: “TSB did not give sufficient consideration to whether a largely single-event migration was the right choice, what the risks would be, or how those risks would be mitigated.” The report acknowledges the appeal honestly — single-event migration is the fastest, cheapest and least complex way to proceed — and then insists that a bank choosing it must understand the risks and robustly test the platform before going live.

Two details matter for anyone tempted to file this under “banks are different.” First, TSB did test. Panorama Consulting's post-mortem notes TSB admitted that while it did perform pilots, they weren't carried out at a sufficient scale — the testing existed, but its scale did not match the scale of the live event. Second, the single-event choice was reportedly made without seriously weighing alternatives, and the board did not adequately probe why the platform remained unstable despite roughly eighteen months of delay and workstreams reported as running around seven months behind schedule. The failure was not a lack of effort. It was concentrating all of the risk into one irreversible moment — a moment regulators later priced at £48.65 million.

To be precise about what this example does and does not prove: it proves the failure mode exists, is expensive, and has been independently dissected. It does not tell you what share of cutovers fail — no dated, published source we reviewed measures that, and this post will not pretend to.

The read-only-first rollout is a specialization of a pattern the industry has endorsed for years. Martin Fowler named it the strangler fig application: grow the new system around the old one, moving behavior across gradually rather than in one event. Fowler is also clear that the pattern is not just a technical trick — it needs four things to work: understand the outcomes you want, decide how to break the problem into smaller parts, deliver those parts successfully, and change the organization's culture so the incremental approach sticks.

Both major cloud vendors document the same pattern independently. Microsoft's Azure Architecture Center defines it as incrementally migrating a legacy system by replacing specific pieces of functionality until the new system comprises all of the old one's features — and, tellingly, its worked example is a database, not a UI: extract domain tables into an isolated domain database, load them via ETL, keep them synchronized with change-data-capture, and only after validation does the new database become the system of record. AWS's Prescriptive Guidance — continuously maintained documentation rather than a dated page — frames the risk driver identically: a big-bang migration approach is risky because of the size and complexity of the monolith, and the pattern exists so users are minimally impacted while the business keeps getting new features.

Microsoft is also candid about when the pattern is wrong: when requests to the back-end can't be intercepted, when you can't modify the legacy system to redirect calls, when the system is small enough that wholesale replacement is simple, or when you must fully decommission the old solution quickly. Those caveats are worth taking seriously — an internal ops app that can't read the incumbent's data at all has no read-only rollout to run. But for the common case — a new operations app growing up next to a CRM, an ERP, or a long-lived spreadsheet that people depend on every day — the read-only-first rollout simply takes that pattern one step further than the migration guides do.

Here is the pattern in full. Build the replacement app against a synchronized copy or a direct read path into the incumbent's data. Ship it to real users on day one — as a read-only surface. Let people live in it: dashboards, search, record views, reports. Meanwhile, log every place where what the new app computes or displays disagrees with what the incumbent shows. Each disagreement is a free correctness test that cost nobody any damage. Only when a specific surface has run clean does it become a candidate to accept writes — and then only that surface.

Which copy of the data the app should read, and how stale a read can safely be, is a real question — it is just a different question, covered in our guide to read paths and stale data; this post assumes reads are solved and asks when the application earns the right to write.

It helps to see what the pattern is not. It is not shadow deployment. Portainer's definition of shadow deployment is a new version running in parallel that receives a copy of live traffic and processes it without ever returning a response to the user — the comparison happens off the request path, in a metrics pipeline or reconstructed from logs. A read-only ops app is close to the inverse: it does answer real users, it just answers only with reads. That inversion quietly avoids most of the failure modes Portainer catalogs for shadow mode: no doubled infrastructure to mirror traffic into, no “side effect leakage” where a shadow call accidentally hits a real payment or notification system, no timestamp and non-determinism noise producing false diffs, and no separate shadow workload inheriting production's regulatory requirements. A read-only app has no side effects to leak by construction.

The deeper property is organizational, and it is the part the migration literature mostly skips: a read-only launch converts the scariest question — “do we trust the new system?” — from an argument into a measurement. Nobody has to assert trust in a meeting. The disagreement log either shrinks to zero on a surface or it doesn't, and everyone can see which.

“Our app doesn't write” is a policy. A database role that cannot write is a guarantee. An application that merely chooses not to issue writes is one ORM default, one migration script, one copy-pasted admin endpoint away from writing anyway — so the read-only phase should be enforced where the application cannot override it: in the database's permission system.

In Postgres, the good news is the starting posture. Crunchy Data's walkthrough puts it plainly: “Postgres is not permissive by default (whew!) so we need to explicitly GRANT some of the privileges.” A genuinely read-only role is built from explicit GRANT SELECT ... TO app_readonly statements against PostgreSQL's GRANT privilege vocabulary — SELECT and nothing else from the write-capable set (INSERT, UPDATE, DELETE, TRUNCATE, and the rest).

The same walkthrough documents the trap in the naive recipe: default privileges only apply to objects created by the role that set them, so if a different user creates a new table, a read-only role granted access at setup time won't be able to read it — and, more dangerously in the other direction, an incompletely-scoped role can quietly drift from what you believed you granted. Postgres 14 added the built-in pg_read_all_data role precisely to close that gap, granting read access to objects created by any user rather than requiring per-creator ALTER DEFAULT PRIVILEGES statements. If you are on 14 or later, use it for the rollout role; if not, re-apply default privileges per creating role and audit.

How clean does a surface's read-only period have to be before it earns a write path? The honest answer is that the tolerance depends on the stakes of the data, and the industry's practitioner guidance reflects that. Data-migration reconciliation guidance commonly sets tolerance at zero for financial and transactional records — reconciled field by field — while lower-priority data domains are checked with aggregate and count-based methods, with non-zero tolerances commonly cited around 0.01%. Treat those figures as practitioner convention, not a regulatory standard; no named standards body publishes a universal number.

Duration guidance from the same practitioner literature is equally unstandardized but directionally consistent: payroll-class systems are commonly run in parallel for two to three full processing cycles before cutover, and financial-servicing implementations are described as using daily transaction matching over roughly fifteen days to three months to build confidence — with the recurring advice that it is better to extend a parallel run than to discover major issues after full cutover. The read-only-first pattern turns that advice from a company-sized decision into a surface-sized one.

The ladder below is an illustrative scenario for a hypothetical operations app — the surface names and numbers are invented for this post, not drawn from TSB or any real client. The shape, not the specific cells, is the point: three write-eligible surfaces entered the read-only period, and exactly one earned a write path in the first month.

Three design choices in the ladder do the real work. First, the test is matched to the stakes: the financial surface gets the zero-tolerance, field-level treatment the reconciliation literature reserves for transactional records; the contact-detail surface needs a clean cycle, not perfection from day zero. Second, there is an intermediate rung between reading and writing — proposing. A surface can draft the write and queue it for a human to approve, which produces exactly the evidence a graduation decision needs. That rung is also where this pattern touches agent governance: approval gates govern what an agent is allowed to do; the ladder governs when the application itself is allowed to write. Third, every rung is per-surface. The failure mode the TSB report describes — one decision, one event, all accounts — cannot be reproduced by a process that is structurally incapable of graduating more than one surface per decision.

The strangler-fig and shadow-mode literature share an unexamined assumption: that the end state is full replacement. Microsoft's pattern ends with the legacy tables removed; AWS's third step is named “eliminate.” For an internal operations app, we think that assumption deserves to be rejected explicitly. Some surfaces should never graduate to write — not because they failed a test, but because the incumbent is genuinely the better place for a human to make that edit.

A finance controller's spreadsheet with fifteen years of embedded institutional logic; a CRM screen an operations team has used daily for a decade, with its validation rules, its audit trail, and its muscle memory — these are often the right editing surface for the people accountable for the data. What they lack is a good view: fast cross-record search, a clean dashboard, a report a manager can open on a phone. A read-only app that provides the view while the incumbent keeps the pen is not an unfinished migration. It is a finished product with a deliberately drawn boundary — and it is a large share of the value of the CRM automation work we ship: the system of record stays authoritative, and the new surface makes it dramatically more usable.

The permanent-read-only decision also caps risk in a way no test can. A surface with no write path has no write bugs, no write permissions to audit, no rollback story to maintain. When the underlying data is high-stakes and the editing workflow in the incumbent works, “a better view, forever” is not a compromise — it is the highest-return, lowest-risk deliverable in the whole program.

This pattern slots into decisions you are probably already making. It picks up after the build-vs-buy decision is made in favor of building: read-only-first is how the thing you chose to build gets adopted without betting the operation on it. It is also a different axis from deployment mechanics — a feature-flag rollout controls when code ships; this ladder controls when the application earns the authority to write. A candidate worked example from our own archive: an agent-audited costing engine is precisely the kind of surface that should compute and display for a long time before any figure it produces flows back into the system of record.

Looking forward, we expect the graduation ladder to become a governance artifact in its own right — a living document that answers “what can the new system touch, and why?” the way the database's grant list answers it technically. Teams that adopt it get a second benefit almost for free: the read-only period produces a disagreement log, and that log is the best requirements document the write phase will ever have. If you are sequencing a replacement like this around a CRM, an ERP, or a long-lived spreadsheet, this staged rollout is exactly the kind of program our AI transformation engagements are built around.