Real-time data and streaming: Kafka, events and the payment event log

A payment is an event, and a payment system is a stream of events. When a customer initiates a payment, an event is created. When the payment is validated, another event. When it is screened, settled, and credited, more events. The event log is the complete history of the payment, in order, with a timestamp for each event. A system built on events can answer any question about a payment by replaying the log, can recover from a failure by replaying the log, and can feed analytics by streaming the log. A system built on a database that only stores the current state cannot do any of these. Kafka is the most common streaming platform for payment events. A payment hub publishes each event to a Kafka topic, and the consumers subscribe to the topics they need: the screening engine subscribes to the initiation topic, the ledger subscribes to the settlement topic, the analytics subscribes to all of them. Kafka persists the events, so a consumer that was down can catch up, and it partitions the topics, so the volume can scale. The requirements must state the topics, the event schema, the partitioning key, and the retention period for each topic. The event schema is the contract. Each event must have a type, a timestamp, a payment identifier, and a payload with the data for that event. The schema must be versioned, because the event format will change, and the consumers must be able to handle both the old and the new version during the transition. The schema must be registered in a schema registry, and the registry must enforce compatibility, so a producer cannot publish an event that breaks the consumers. A schema change without a compatibility check will break the system in production. Exactly-once processing is the hardest problem in streaming. A payment event must be processed exactly once, not zero times and not twice, because processing it twice will double-pay. Kafka provides at-least-once delivery, which means a consumer may receive the same event twice, and the consumer must therefore be idempotent: processing the same event twice must have the same effect as processing it once. The idempotency is usually achieved by storing the event identifier and checking it before processing. The requirements must state the idempotency mechanism for each consumer, and the test must prove that a duplicate event does not cause a duplicate payment. The payment event log as a source of truth is an architectural decision with deep consequences. If the log is the source of truth, the current state of a payment is derived by replaying the log, and the database is a cache, not the truth. This makes recovery simple, because the state can always be rebuilt, and makes auditing simple, because the log is the audit trail. But it also means the log must be immutable, must be retained for the statutory period, and must be protected, because a log that can be altered is not a source of truth. The requirements must state the retention, the immutability, and the access control for the log. Streaming analytics is the real-time use of the event log. A fraud model that subscribes to the initiation topic can score a payment before it is processed. A liquidity forecast that subscribes to the settlement topic can update the forecast as payments settle. An operations dashboard that subscribes to the exception topic can show the exceptions as they happen. The analytics must be designed around the events, not around a batch extract, because a batch that runs once a day cannot score a payment that settles in seconds. For a business analyst, the streaming architecture is a set of requirements: the topics, the schemas, the partitioning, the retention, the idempotency, and the consumers. The acceptance criteria must include a test where a payment is traced through the event log, event by event, and the state at each point is verified. Last reviewed 7 November 2026. Further reading: the Apache Kafka documentation; Martin Kleppmann, Designing Data-Intensive Applications; the Confluent documentation on event-driven architecture; the Bank for International Settlements work on real-time payment data.