Observability: metrics, logs, traces and alerting for payments

Observability is the ability to answer, for any payment at any time, where it is, what happened to it, and why. A payment engine that is not observable is a black box: when a customer calls to ask where their payment is, the bank cannot answer, and when a payment fails, no one knows which step failed. Observability has three pillars, metrics, logs and traces, and a payments SME must understand all three because they are the difference between an engine that can be operated and one that cannot. Source: observability practices for payment platforms; bank payment operations monitoring standards. Metrics are numbers that describe the health of the engine over time. The essential payment metrics are throughput (payments per minute, by engine and by clearing system), latency (time from intake to settlement, by percentile, not just average), success rate (the proportion of payments that complete without exception), and queue depth (how many payments are waiting in each worklist). For a business analyst, the requirement is to define this behaviour explicitly, because the engine does what the requirement says and nothing more. A latency average of 4 seconds hides the truth if the 99th percentile is 45 seconds, because the customers whose payments took 45 seconds are the ones who complain. Queue depth is the earliest warning: if the repair worklist grows from 20 items to 200 items in an hour, something upstream is producing bad instructions, and the team needs to know before the queue becomes unmanageable. Metrics are aggregated over time and shown on dashboards that the operations team watches continuously. Logs are the detailed record of what the engine did, one entry per event. Every state change, every message sent and received, every validation result and every operator action writes a log entry with a timestamp, the payment identifier, the step, and the outcome. Logs are the raw material for investigation: when a payment of EUR 25,000 is returned with reason code AC04, the operator reads the log to find the exact moment the beneficiary account was found to be closed, and which system reported it. Logs must be structured, with fields rather than free text, so they can be searched and correlated. A log entry that says only payment failed is useless; one that says payment PAY-20261105-8841, step enrichment, field beneficiary IBAN, result invalid, code AC01 is actionable. Traces follow a single payment across every system it touches. A trace assigns a correlation identifier at intake, and every system that handles the payment carries that identifier in its logs and in the messages it sends. When a cross border payment passes through the hub, the correspondent bank, the clearing system and the beneficiary bank, the trace links the log entries in each system into one timeline. This is how an operator answers the question where is my payment: they open the trace, see that the payment left the hub at 10:04, reached the correspondent at 10:06, and is waiting for the beneficiary bank to credit the account. Without a trace, the operator must call each system separately and piece the story together by hand, which takes hours instead of seconds. Alerting turns metrics into action. An alert fires when a metric crosses a threshold: queue depth above 100, success rate below 99 percent, latency 99th percentile above 30 seconds, or a clearing system connection down for more than 60 seconds. Each alert has a severity, an owner, and a runbook: a written procedure that tells the on call engineer exactly what to check and what to do. The operations team must be able to see this state on a dashboard, because a payment that cannot be seen cannot be managed. A severity one alert, such as the clearing system connection being down, pages the on call engineer immediately; a severity three alert, such as a slow rise in queue depth, creates a ticket for the next working day. An alert without a runbook is noise, and a runbook that has never been tested is a hope, not a control. For a business analyst, the requirement is to define the alerts, the thresholds, the severity, the owner and the runbook for every payment type the engine carries. Dashboards turn metrics into a picture the operations team can act on. A payment operations dashboard shows throughput, latency, success rate and queue depth for each engine and each clearing system, updated every minute. It also shows the payments that are currently stuck, the worklists that are growing, and the alerts that are active. The dashboard is the first thing the operations team looks at in the morning and the first thing they check when a customer calls. A dashboard that shows only averages hides the problems; a dashboard that shows percentiles, queue depths and stuck payments reveals them. For a business analyst, the requirement is to define what the dashboard shows, how often it updates, and who watches it at each hour of the day. Capacity alerts deserve special attention, because a capacity problem that is not caught early becomes an outage. The engine must alert when CPU, memory, database connections or thread pools approach their limits, not when they are exhausted. An alert at 80 percent utilisation gives the team time to scale out or to shed load; an alert at 100 percent utilisation is a report that the engine has already stopped. The runbook for a capacity alert must state what to scale, how to scale it, and how to verify that the scaling worked. A bank that processes 250 million payments a year cannot afford to discover at 09:00 on the last working day of the month that its engine cannot handle the payroll run. Further reading: observability practices for payment platforms; Site Reliability Engineering, Google. Last reviewed 7 November 2026.