How Apache Kafka works? Kafka Series - Part 1

what is kafka?

Firstly, Kafka is a streaming platform which follows the pub/sub pattern for sending the messages/monitor the events

Usecase of kafka

Kafka is generally used for two broad classes of applications:

Building real-time streaming data pipelines that reliably get data between systems or applications

Building real-time streaming applications that transform or react to the streams of data

Kafka Core API'S :

Producer API : it allows an application to publish a stream of records to a particular topic. we will see what topic is in the later part of this article.
Consumer API : it allows an application to subscribe to one or more topic and process the stream of records produced to them
Streams API : allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics.
Connector API : allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems.

How kafka works ?

In layman terms, Producer pushes the records into Kafka broker where consumer take those records and take action according to that message/event.

Kafka Architectural Overview

what is Topic and Partition ?

Topic in Kafka is nothing but a feed where a producer can push the records and consumer can get the records. Topics in Kafka are always multi-subscriber; that is, a topic can have zero, one, or many consumers that subscribe to the data written to it.

To make a topic fault tolerant and to handle the throughput, topic is maintained in different partitions. partitions can be in the same disk or in different disk.

What is Kafka Broker ?

Kafka Broker contains the cluster. Each Kafka Broker contains the topic log partitions. Connecting to one broker starts a client to the entire Kafka cluster. For failover, you want to start with at least three to five brokers. A Kafka cluster can have, 10, 100, or 1,000 brokers in a cluster if needed.