Open Access

A Critical Analysis of Apache Kafka's Role in Advancing Microservices Architecture: Performance, Patterns, and Persistence

pdf

Anh N. Tran ¹ , Siew H. Lim ¹ ,

⁴ Faculty of Computer Science, University of Technology, Ho Chi Minh City, Vietnam

⁴ School of Computing, National University of Singapore, Singapore

Abstract

Purpose: This article critically analyzes the strategic adoption of Apache Kafka as a foundational event streaming framework within Microservices Architecture (MSA), evaluating its impact on system performance, architectural design, and operational complexity in modern distributed computing.

Methodology: The research synthesizes academic literature and industry best practices, detailing Kafka’s distributed log architecture (brokers, topics, partitions) and its alignment with Event-Driven Architecture (EDA) principles. A systematic review is conducted on key microservices patterns—Event Sourcing, Saga, and CQRS—to model inter-service communication and distributed data consistency. The study also investigates the empirical trade-offs associated with performance tuning and system governance.

Findings: Kafka provides an essential backbone for achieving high-throughput, low-latency, and decoupled services, empirically handling millions of events per second. The distributed log structure inherently supports complex patterns necessary for distributed data management, such as the Saga pattern for transactional integrity. However, its adoption introduces significant operational overhead related to schema evolution management, the complexities of achieving eventual consistency, and the necessity for robust distributed observability solutions like tracing and correlated logging.

Originality: This work offers a comprehensive framework for design and deployment, moving beyond basic integration to emphasize the challenges of governance and stateful stream processing, thereby supporting the strategic architectural decisions required for an 8000+ word manuscript.

Keywords

Event-Driven Architecture, Microservices, Apache Kafka, Distributed Systems

References

B. R. Hiraman, C. Viresh M. and K. Abhijeet C., "A Study of Apache Kafka in Big Data Stream Processing," 2018 International Conference on Information , Communication, Engineering and Technology (ICICET), 2018, pp. 1-3, doi: 10.1109/ICICET.2018.8533771.

R. Shree, T. Choudhury, S. C. Gupta and P. Kumar, "KAFKA: The modern platform for data management and analysis in big data domain," 2017 2nd International Conference on Telecommunication and Networks (TEL-NET), 2017, pp. 1-5, doi: 10.1109/TELNET.2017.8343593.

Kesarpu, S., & Hari Prasad Dasari. (2025). Kafka Event Sourcing for Real-Time Risk Analysis. International Journal of Computational and Experimental Science and Engineering, 11(3). https://doi.org/10.22399/ijcesen.3715

Shaheen, J. A.. “Apache Kafka: Real Time Implementation with Kafka Architecture Review.” International journal of advanced science and technology 109 (2017): 35-42.

Dobbelaere, P., &Esmaili, K.S. (2017). Kafka versus RabbitMQ. ArXiv, abs/1709.00333.

P. Le Noac'h, A. Costan and L. Bougé, "A performance evaluation of Apache Kafka in support of big data streaming applications," 2017 IEEE International Conference on Big Data (Big Data), 2017, pp. 4803-4806, doi: 10.1109/BigData.2017.8258548.

B. Yadranjiaghdam, N. Pool and N. Tabrizi, "A Survey on Real-Time Big Data Analytics: Applications and Tools," 2016 International Conference on Computational Science and Computational Intelligence (CSCI), 2016, pp. 404-409, doi: 10.1109/CSCI.2016.0083.

Singh, V. (2025). Securing Transactional Integrity: Cybersecurity Practices in Fintech and Core Banking. QTanalytics Publication (Books), 86–96. https://doi.org/10.48001/978-81-980647-2-1-9

Sayyed, Z. (2025). Development of a Simulator to Mimic VMware vCloud Director (VCD) API Calls for Cloud Orchestration Testing. International Journal of Computational and Experimental Science and Engineering, 11(3). https://doi.org/10.22399/ijcesen.3480

International Journal of Modern Computer Science and IT Innovations

A Critical Analysis of Apache Kafka's Role in Advancing Microservices Architecture: Performance, Patterns, and Persistence

Abstract

Keywords

References

Most read articles by the same author(s)

Similar Articles