Apache Kafka

This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these messages) This article is written like a manual or guide. Please help rewrite this article and remove advice or instruction. (November 2023) This article may rely excessively on sources too closely associated with the subject, potentially preventing the article from being verifiable and neutral. Please help improve it by replacing them with more appropriate citations to reliable, independent, third-party sources. (November 2023) (Learn how and when to remove this message) (Learn how and when to remove this message)

Apache Kafka^[1]

Original author(s)	LinkedIn
Developer(s)	Apache Software Foundation
Initial release	January 2011; 13 years ago (2011-01)[2]
Stable release	3.9.0[3]  / 6 November 2024
Repository	github.com/apache/kafka
Written in	Scala, Java
Operating system	Cross-platform
Type	Stream processing, Message broker
License	Apache License 2.0
Website	kafka.apache.org

Apache Kafka is a distributed event store and stream-processing platform. It is an open-source system developed by the Apache Software Foundation written in Java and Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Kafka can connect to external systems (for data import/export) via Kafka Connect, and provides the Kafka Streams libraries for stream processing applications. Kafka uses a binary TCP-based protocol that is optimized for efficiency and relies on a "message set" abstraction that naturally groups messages together to reduce the overhead of the network roundtrip. This "leads to larger network packets, larger sequential disk operations, contiguous memory blocks [...] which allows Kafka to turn a bursty stream of random message writes into linear writes."^[4]