Resilience at Scale: Transactions, Breakers and Recovery

Distributed transactions are the hardest problem in hub design, because a payment touches many systems that cannot share one database transaction. A classic two phase commit across the hub, core banking and a clearing system would lock resources, fail slowly and couple every system to every other. Modern hubs therefore use the saga pattern, where each step has a compensating action that undoes it if a later step fails. If settlement fails after the customer account was debited, the saga runs a compensation that credits the account back and marks the payment as failed. The design must record every step durably, so a crash halfway through a saga can be resumed or compensated without losing money. Circuit breakers stop a failing dependency from dragging the whole hub down with it. A circuit breaker monitors calls to a service, such as the sanctions screening provider, and counts failures over a rolling window. If failures pass a threshold, for example 50 percent of calls failing in 60 seconds, the breaker opens and calls fail fast instead of waiting for a timeout. The hub can then queue payments for screening, use a cached result where policy allows, or route to a fallback provider. After a cool down period, the breaker allows a few trial calls, and closes again only when the service recovers. Without breakers, one slow vendor can exhaust every thread in the hub. High availability means designing the hub so that no single failure stops payments, and it starts with removing single points of failure. Each component runs on at least two instances across two availability zones, databases replicate synchronously to a standby, and load balancers route around failed instances in seconds. A well designed hub targets 99.95 percent availability, which allows about 22 minutes of downtime per month, and measures it from the customer perspective, not from a single server. Health checks must test real dependencies, such as a database write and a screening call, because a service that reports healthy while its database is down is worse than one that reports failure. Disaster recovery is the plan for losing a whole data centre, and it is measured with two numbers every analyst should know. Recovery time objective is how long recovery may take, and recovery point objective is how much data may be lost. A large bank typically sets a recovery time objective of 2 hours and a recovery point objective of 5 minutes for its payment hub, with a hot standby site that can take over automatically. That means at most 5 minutes of payment data may need to be replayed, and the hub must be back within 2 hours. These targets are tested with full failover exercises at least twice a year, because an untested recovery plan is only a document. Volumes and performance budgets decide whether the architecture is real or only a diagram. A large retail bank hub may process 3 million payments a day, with peaks of 1,200 payments per second at 9 in the morning when salary files and standing orders fire together. End to end processing for a domestic instant payment must complete in under 5 seconds, while a cross border SWIFT payment may allow 30 seconds for screening and routing before the customer sees a sent status. Each component needs its own latency budget, for example 200 milliseconds for fraud scoring and 500 milliseconds for sanctions screening, so that the total stays inside the customer promise even at peak load. Resilience must be proven, not assumed, and that means deliberate failure testing in production like conditions. Chaos testing introduces controlled failures, such as killing a fraud engine instance, delaying a core banking response by 3 seconds, or cutting connectivity to one availability zone, and then measures whether payments still complete and whether alerts fire correctly. A mature hub runs these tests monthly and tracks two results, namely the percentage of payments that completed without manual intervention and the time to detect each injected failure. If a test shows that a single database failure blocks all payments for 20 minutes, the design has a flaw that a diagram would never reveal.