System design deep dive: payment APIs at scale

The API gateway is the front door for all payment APIs, and it handles routing, security, and traffic management before any request reaches a backend service. A request arrives at the gateway, which validates the token, checks the rate limit, logs the call, and routes it to the correct service based on the path and version. The gateway runs as a cluster of stateless nodes behind a load balancer, so it scales by adding nodes, and a large bank's gateway handles 10,000 to 20,000 API calls per second at peak. The gateway also terminates TLS, enforces request size limits, and blocks malformed requests early, so backend services only see clean, authenticated traffic and can focus on payment processing. Authentication for payment APIs uses OAuth 2.0 for customer facing access and mutual TLS for partner and corporate access. With OAuth, a client obtains a short lived access token, typically valid for 15 to 60 minutes, and presents it with each request, while refresh tokens allow long lived sessions without storing passwords. With mutual TLS, both the client and the server present certificates, so the bank knows exactly which system is calling, and certificates are rotated at least once a year. Every API key and certificate is tied to a specific client and a specific set of permissions, so a client that can read account balances cannot also initiate payments unless it has been explicitly granted that scope. Rate limiting protects the platform from overload and from abuse. Limits are set per client, per API, and per endpoint, for example 100 requests per second for payment initiation and 1,000 per second for balance queries, with a burst allowance of 20 percent above the steady rate. When a client exceeds its limit, the gateway returns a 429 response with a retry after header, and the client is expected to back off rather than retry immediately. Limits are enforced in a distributed store so they work correctly across all gateway nodes, and they are reviewed monthly against actual usage. A client that repeatedly exceeds its limit is contacted by the API team, because sustained overuse usually means a bug or a misunderstanding rather than malice. Idempotency keys make payment APIs safe to retry. When a client creates a payment, it sends a unique idempotency key with the request, and the bank stores that key with the payment record. If the same key arrives again, whether because the first response was lost or the client retried too quickly, the bank returns the original result instead of creating a second payment. Keys are stored for at least 24 hours, and often for seven days, so retries over a weekend are still safe. This simple mechanism prevents the most expensive API failure, a duplicate payment, and it is a requirement for any production payment API, not an optional extra. Webhook delivery and versioning keep clients informed without polling. When a payment changes state, the bank sends a webhook event to the client's registered URL, with a signed payload so the client can verify it came from the bank. If delivery fails, the bank retries with exponential backoff, for example after 1 minute, 5 minutes, 15 minutes, and 1 hour, for up to 24 hours, and then moves the event to a dead letter queue for manual review. API versioning uses the URL path, such as v1 and v2, so old clients keep working while new clients adopt new features. Every API call is written to an audit log with the client, the endpoint, the timestamp, the request identifier, and the outcome, and the log is retained for at least seven years.