Basics of Kafka Architecture

Kafka is a publish-subscribe based durable messaging system exchanging data between processes, applications, and servers

Key components/terminologies in Kafka Architecture:

Producers, Topic, Consumers, Broker ,Consumer Group, Partitions, Offset ,ZooKeeper ,Replications ,Leader, Kafka API’s

Kafka basic flow:

Kafka producers write to topics, while Kafka consumers read from topics. Topics represent commit log data structures stored on disk. Kafka adds records written by producers to the ends of those topic commit logs. Topic logs are also made up of multiple partitions, straddling multiple files and potentially multiple cluster nodes. Consumers can use offsets to read from certain locations within topic logs. Consumer groups each remember the offset that represents the place they last read from a topic.

Partitions of topic logs are distributed across cluster nodes, or brokers, to achieve horizontal scalability and high performance. Kafka architecture can be leveraged to improve upon these goals, simply by utilizing additional consumers as needed in a consumer group to access topic log partitions replicated across nodes. This enables Apache Kafka to provide greater failover and reliability while at the same time increasing processing speed.