Apache Kafka for Real-Time Data Integration: A Scalability and Performance Assessment
Year: 2026
Type: conference-paper
Abstract: Modern card-issuer platforms must authorize millions of payments per day within strict response-time budgets while keeping ledgers and account balances consistent. Apache Kafka is widely used to decouple durable balance maintenance from the latency-critical authorization path, yet quantitative evidence on how its throughput, latency, and resource usage scale under realistic payment workloads remains fragmented across the literature. This paper does a few things for Kafka. First it creates a test to see how well a six-broker Kafka cluster works. This test uses a stream of transactions that're like the ones that happen on the MasterCard Network. Each transaction is checked for fraud by the FICO Falcon service. Then sent to Kafka. Kafka then updates the account balances for each transaction. The paper also looks at how long it takes for transactions to go through the system. It breaks down this time into three parts: the time it takes for the producer to send the transaction the time it takes for the broker to process the transaction and the time it takes for the consumer to receive the transaction. The paper compares two ways to make the system work better: adding brokers or making each broker more powerful. It finds that the biggest problem is that the disks that store the data get too busy when there are copies of the data. The tests were run with numbers of transactions per second: 50,000, 200,000 and 500,000. The results show that adding brokers makes the system work better but only up, to a point. The time it takes for transactions to go through the system stays very low even when there are copies of the data. The computers that run the system do not get too busy. The disks that store the data do. Kafka is used by companies that issue cards like MasterCard. The results of this paper will help these companies make their systems work better. It will also help people who work on Kafka to make it work better in the future. The paper shows that Kafka can work well even with a lot of transactions happening at the same time.
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7203694077