Monzo built a full backup banking platform on GCP

Monzo built a full backup banking platform on GCP

Monzo has built Monzo Stand-in, a backup banking infrastructure that runs on Google Cloud Platform (GCP) as a fallback for its Primary Platform, which runs on Amazon Web Services (AWS). Stand-in is not the company's main way of delivering reliability; it is described as a backup of last resort, an extra line of defence for when unforeseen incidents cause outages that Monzo's day-to-day engineering effort cannot prevent. Stand-in covers the features Monzo considers most important: spending on cards, withdrawing cash, sending and receiving bank transfers, checking balances and transactions, and freezing or unfreezing cards. When it is switched on, the Monzo App detects this in the background and flips to a simplified interface built around those functions. Both platforms run independently. Each is built on Kubernetes clusters with its own database, queueing and locking systems, and the services that run in Stand-in never run in Primary, or vice versa, even for behaviour common to both, such as processing a card payment. Each platform can independently approve or decline a transaction and connect to payment networks through multiple physical data centres. Monzo chose this duplicated-but-different design deliberately rather than running identical services on both platforms. Running the same services on both would require strong data consistency, meaning a write would only count as successful once it landed on both platforms; if either platform went down, Monzo would be unable to write data at all without breaking that consistency, which would cut availability rather than protect it. Using different software on the two platforms also reduces the chance that a single bug affects both at once. Monzo notes that cloud providers such as AWS, GCP and Azure have largely solved for hardware failure, so the more likely cause of an outage today is a bug in Monzo's own code or processes, which independent codebases help contain. Data moves from Primary to Stand-in through a Stand-in Data Syncer, which consumes a subset of Primary's event stream to keep Stand-in supplied with the minimal state it needs: balances, limited transaction history, card and account details, and information like pots and payees. This data is treated as immutable once synced, the sync is eventually consistent rather than strictly consistent, and Monzo monitors the sync lag and alerts if it grows too large. Tokenised data such as card PANs follows a similar but separate flow, exchanged between the two platforms' tokenisation systems using different encryption keysets. When Stand-in itself makes decisions while active, those outcomes are recorded in a durable queue as Monzo Advices and are only authoritative for as long as Stand-in is running; Primary remains the system of record and is expected to apply the effects of these Advices verbatim once it can process them again. Because applying an Advice verbatim can happen against a Primary balance that has since diverged, a customer could in rare cases end up in an unapproved overdraft; Monzo says this is extremely unlikely in practice and runs additional controls against it. To stop the same payment being counted twice once Primary catches up and re-syncs it back to Stand-in, the two platforms tag transactions with a shared Correlation ID and merge the matching records. Turning Stand-in on is a manual step, coordinated through a Stand-in Configuration service that runs in both platforms and currently triggered by engineers using CLI tooling when they detect an outage in a critical service; Monzo says this could be automated further using the same heuristics engineers already apply by hand. Turning it back off is also a deliberate, manual choice, so traffic is rolled back to Primary gradually rather than switching over all at once the instant Primary's API recovers. For payments specifically, Monzo first routes traffic through Primary, which proxies it to Stand-in, giving fine control over which customers or how much traffic shifts. If Primary is completely unreachable, Stand-in can instead connect directly to payment networks through Monzo's own data centres, a blunter option with far less control over routing but one Monzo says it needs to stay resilient. Both routes are tested continuously in production. Running Stand-in this way costs around 1% of what the Primary Platform costs, a figure Monzo expects to rise only marginally during an actual incident; replicating all of Primary's systems and data instead, with full consistency, would risk doubling total platform costs. The post states that Monzo customers may already have experienced Stand-in during a major platform incident in August 2024 that affected most of its systems, but the available text cuts off before describing what happened during that incident.

Key facts

  • Monzo Stand-in is a fully separate banking platform on Google Cloud Platform, backing up the Primary Platform that runs on Amazon Web Services, and it deliberately runs different software rather than copies of Primary's services.
  • Stand-in covers only the core features Monzo sees as essential: card spending, cash withdrawal, bank transfers, balance and transaction checks, and freezing or unfreezing cards, switching the Monzo App to a simplified interface when active.
  • Data flows one way from Primary to Stand-in as immutable, eventually consistent state via a Stand-in Data Syncer; effects created while Stand-in is active are queued as Monzo Advices for Primary to apply verbatim once it recovers, with Correlation IDs preventing transactions from being counted twice.
  • Running Stand-in in the background costs about 1% of the Primary Platform's cost, versus a potential doubling of total platform costs if Monzo instead replicated all systems and data with strong consistency.
  • Enabling and disabling Stand-in are manual, deliberate decisions made by engineers via CLI tooling, and payments can route through Primary proxying to Stand-in or, if Primary is fully down, directly from Stand-in to payment networks through Monzo's own data centres.

Why it matters

The post is a rare first-person account of how a regulated retail bank engineers for total cloud-provider failure rather than just hardware or availability-zone failure. Monzo's central point is that modern cloud platforms have mostly solved hardware failure, so the realistic remaining risk is software: a bug in your own code or process. Its answer is architectural independence rather than duplication: a separate cloud vendor, a separate codebase, and acceptance of eventual consistency in exchange for the two platforms never sharing a single point of failure.

Who it affects

Directly, Monzo's retail banking customers, whose cards, transfers and balance checks keep working during an outage that would otherwise take Monzo fully offline. Indirectly, engineers and architects at other financial or otherwise availability-critical companies designing disaster recovery, who get a concrete worked example of trading strong consistency for availability and of running a second platform on a second cloud vendor.

How to use it

There is no product or price here; this is Monzo's own infrastructure, not something bought or licensed. Teams designing similar disaster recovery can take the pattern rather than the code: keep the backup platform's feature set deliberately minimal, use a different provider and a different codebase from the primary system, sync only the minimal state needed as immutable data, and make both activation and deactivation manual, deliberate engineering decisions rather than automatic failover.

How solid is it

This is Monzo's own engineering blog describing its own production system, so the claims are not independently verified, but they are specific and operational rather than promotional: exact figures for relative cost (about 1% of Primary), the naming and function of internal components (Stand-in Data Syncer, Monzo Advices, Correlation IDs), and a statement that both payment-routing paths are tested continuously in production. The source text available for this retelling cuts off mid-sentence describing an August 2024 incident, so no detail on that incident's scope, duration or how Stand-in was used during it is available here.

Risks and caveats

By design, Stand-in operates on an eventually consistent, not strictly consistent, view of a customer's balance; Monzo's own text acknowledges that applying its queued Advices verbatim back into the Primary Platform could, in rare cases, approve a transaction that should have been declined, potentially pushing a customer into an unapproved overdraft, though it says this is extremely unlikely and adds controls against it. The system also depends on human judgment: engineers currently decide by hand when to enable and disable Stand-in, and Monzo notes this manual process could be automated further but has not been yet.

“Today it doesn't matter how many data centres you have if you run the same software in them all.”

— Monzo engineering blog