How Apache Kafka works? Kafka Series - Part 1

what is kafka?

Firstly, Kafka is a streaming platform which follows the pub/sub pattern for sending the messages/monitor the events

Usecase of kafka

Kafka is generally used for two broad classes of applications:

  • Building real-time streaming data pipelines that reliably get data between systems or applications
  • Building real-time streaming applications that transform or react to the streams of data

Kafka Core API'S :

  • Producer API : it allows an application to publish a stream of records to a particular topic. we will see what topic is in the later part of this article.
  • Consumer API : it allows an application to subscribe to one or more topic and process the stream of records produced to them
  • Streams API : allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics.
  • Connector API : allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems.

How kafka works ?

In layman terms, Producer pushes the records into Kafka broker where consumer take those records and take action according to that message/event.

kafka architecture

Kafka Architectural Overview

what is Topic and Partition ?

Topic in Kafka is nothing but a feed where a producer can push the records and consumer can get the records. Topics in Kafka are always multi-subscriber; that is, a topic can have zero, one, or many consumers that subscribe to the data written to it.

log anatomy

To make a topic fault tolerant and to handle the throughput, topic is maintained in different partitions. partitions can be in the same disk or in different disk.

What is Kafka Broker ?

Kafka Broker contains the cluster. Each Kafka Broker contains the topic log partitions. Connecting to one broker starts a client to the entire Kafka cluster. For failover, you want to start with at least three to five brokers. A Kafka cluster can have, 10, 100, or 1,000 brokers in a cluster if needed.

kafka zookeeper

Kafka Broker Overview

To Configure the Apache Kafka in local machine and Run the server. Kafka Series Part 2

To Read More

Building Nodejs Microservice - A Cl...

This Article explains everything about how to build Nodejs Microservices in clou...

I Accidentally wiped the entire dat...

One of the tragic accident in my job turned out to be good learning for me in re...

List of Docker Container Commands y...

This article covers list of commands that you should know to manage docker conta...