Orchestration failure handling: timeouts, retries, compensation and sagas

A payment crosses many systems, and any one of them can fail, time out or return an ambiguous result. Orchestration is the discipline of making the whole journey reliable even though each step is unreliable. The engine does not assume a step succeeded because it sent a request; it assumes nothing until it has a positive confirmation, and it has a defined recovery for every way a step can fail. The three failure modes a payments SME must be able to explain are the timeout, the duplicate and the partial failure, and the three recovery patterns are retry with idempotency, compensation, and the saga. Source: microservices and distributed systems patterns as applied to payment engines; ISO 20022 status reporting. A timeout is the hardest failure because the engine does not know what happened. It sent a pacs.008 to the clearing system and received no response within the agreed window. The payment may have been accepted, rejected, or never received. The engine must not simply resend, because a resend could create a duplicate payment. Instead it enquires: it sends a status request, using the original end to end identification, and waits for the clearing system to confirm the payment status. If the clearing system confirms the payment was accepted, the engine moves on. If it confirms the payment was never received, the engine resends safely. If the clearing system cannot confirm either way, the payment goes to an exception worklist for an operator to resolve with the clearing system directly. The rule is absolute: never resend a payment whose outcome is unknown. For a business analyst, the requirement is to define this behaviour explicitly, because the engine does what the requirement says and nothing more. Retry with idempotency is the recovery for a failure where the outcome is known to be negative. If the clearing system returns a clear rejection, or if a downstream system returns a 500 error and the engine knows the request was not processed, the engine retries the same request with the same idempotency key. The idempotency key is a unique value the engine generates for the request; the receiving system stores it, and if it sees the same key again it returns the original response instead of executing twice. This is what makes a retry safe. A retry without an idempotency key is a duplicate payment waiting to happen. For an API, the client supplies the idempotency key in a header; for an internal engine step, the engine generates it from the payment identifier and the step name. Compensation reverses a step that succeeded when a later step fails. Suppose the engine debits the customer account, then the clearing system rejects the payment because the beneficiary bank is unreachable. The debit has already happened, so the engine must compensate: it credits the customer account back, posts a reversal entry, and notifies the customer. Compensation is not a simple undo, because the original debit may have already been reported to the customer or used in a balance calculation. The engine therefore posts an explicit reversal entry, linked to the original, so the ledger shows both the debit and its reversal and the audit trail is complete. A compensation that is not recorded is a reconciliation break. A saga is the pattern that coordinates a sequence of steps where each step has its own compensation. A cross border payment is a saga: debit the customer, send the instruction to the correspondent, wait for the cover, confirm settlement, credit the beneficiary. Each step has a compensating action, and the saga tracks which steps have completed so it knows which compensations to run if a later step fails. If the correspondent rejects the cover after the customer has been debited, the saga compensates by crediting the customer and notifying the channel. The saga pattern is what lets a payment engine handle a failure at any point in a multi system journey without losing money or leaving the payment in an unknown state. For a business analyst, the requirement is to define the saga for each payment type: the steps, the compensation for each step, and the state in which the payment rests while the saga waits. The recovery for a duplicate is different from the recovery for a timeout, because a duplicate means the money has already moved twice. When the engine detects that a payment was sent twice, it must stop the second payment if it has not yet settled, or recall it if it has. The detection comes from the idempotency key, from the end to end identification, or from the reconciliation process that finds two identical payments. The engine then raises an exception with both payment identifiers, so the operator can see which one is the original and which one is the duplicate, and can confirm the recall of the duplicate. A duplicate that is not detected is a customer who has been debited twice, and a customer who has been debited twice is a complaint, a refund, and a loss of trust. For a business analyst, the requirement is to define the duplicate detection rules, the recall procedure, and the customer notification for a duplicate payment. Testing failure handling is as important as building it, and it is often neglected. A payment engine that has never been tested with a downstream system that times out will behave unpredictably the first time it happens in production. The test pack must include a test for every failure mode: a clearing system that returns no response, a clearing system that returns a 500 error, a database that is unavailable during posting, and a message that arrives twice. Each test must verify that the engine recovers correctly, that no duplicate is created, that the ledger balances, and that the operator can see what happened. A test that only checks the happy path is not a test of the engine; it is a test of the demo. Further reading: microservices patterns, saga pattern; ISO 20022 status and investigation messages. Last reviewed 7 November 2026. The operations team must be able to see this state on a dashboard, because a payment that cannot be seen cannot be managed.