Retries, Dead Letter Queues, Timeouts and Unknown Outcomes

A retry policy defines how many times a payment is sent again, how long to wait between attempts, and how the wait grows with each failure. The usual pattern is exponential backoff, where the first retry waits a few seconds, the next waits longer, and later retries wait minutes, so a struggling system is not flooded with repeated attempts. Each retry must carry the same idempotency key and the same UETR, so the receiving system recognises it as the same payment rather than a new one. In the worked example, Meridian Bank sends the USD 120,000 payment to the correspondent, receives no acknowledgement within thirty seconds, waits one minute and retries, then waits two minutes and retries again. Testing the policy means forcing a timeout, counting the retries, checking the gaps between them, and confirming the key and the UETR stay the same on every attempt. Retry must stop at a defined limit, because endless retry can turn a small failure into a flood and can pay a payment twice if the original attempt actually succeeded. The limit is usually a number of attempts or a total elapsed time, and once it is reached the payment moves to a failed state and the operations team is alerted. The limit must be short enough to protect the customer from long delays, but long enough to ride out a brief outage, and the choice is agreed with the business rather than left to a developer default. In the worked example, Meridian Bank allows three attempts over ten minutes, and after the third failure the USD 120,000 payment is marked failed and routed to the exception queue. Testing the limit means forcing repeated failures, confirming the payment stops after exactly three attempts, and confirming the alert reaches operations. A dead letter queue holds messages that could not be processed after retries, and it is a holding area rather than a bin, because every message in it still needs a decision. Each entry carries the message, the error that stopped it, the number of attempts, and the time it failed, so an operator can see why it stopped and what to do next. Common causes are a malformed message, a missing account, a system that is down, or a validation rule that rejects the content. In the worked example, the USD 120,000 payment lands in the dead letter queue when the correspondent system rejects the message format, and an operator corrects the field, replays the message, and watches it complete. Testing the queue means sending a bad message, confirming it lands in the queue with its error, replaying it after a correction, and confirming the replay creates one payment. A timeout leaves the outcome unknown, because the payment may have been processed by the receiving system even though the acknowledgement never arrived. The golden rule is never to retry blindly, because a blind retry can pay the same USD 120,000 twice. Instead, operations checks the real outcome first, by asking the correspondent for the payment status using the UETR, by checking the settlement account for a matching debit, and by reading the latest status report from the chain. In the worked example, Meridian Bank times out on the correspondent, queries the status by UETR, learns the payment was never received, and only then retries safely. If the query had shown the payment was already settled, operations would have marked it settled and stopped. Testing this flow means forcing a timeout, confirming the system queries before it retries, and confirming no duplicate payment is created in either outcome.