How Apache Kafka works? Kafka Series - Part 1

what is kafka?

Firstly, Kafka is a streaming platform which follows the pub/sub pattern for sending the messages/monitor the events

Usecase of kafka

Kafka is generally used for two broad classes of applications:

  • Building real-time streaming data pipelines that reliably get data between systems or applications
  • Building real-time streaming applications that transform or react to the streams of data

Kafka Core API'S :

  • Producer API : it allows an application to publish a stream of records to a particular topic. we will see what topic is in the later part of this article.
  • Consumer API : it allows an application to subscribe to one or more topic and process the stream of records produced to them
  • Streams API : allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics.
  • Connector API : allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems.

How kafka works ?

In layman terms, Producer pushes the records into Kafka broker where consumer take those records and take action according to that message/event.

kafka architecture

Kafka Architectural Overview

what is Topic and Partition ?

Topic in Kafka is nothing but a feed where a producer can push the records and consumer can get the records. Topics in Kafka are always multi-subscriber; that is, a topic can have zero, one, or many consumers that subscribe to the data written to it.

log anatomy

To make a topic fault tolerant and to handle the throughput, topic is maintained in different partitions. partitions can be in the same disk or in different disk.

What is Kafka Broker ?

Kafka Broker contains the cluster. Each Kafka Broker contains the topic log partitions. Connecting to one broker starts a client to the entire Kafka cluster. For failover, you want to start with at least three to five brokers. A Kafka cluster can have, 10, 100, or 1,000 brokers in a cluster if needed.

kafka zookeeper

Kafka Broker Overview

To Configure the Apache Kafka in local machine and Run the server. Kafka Series Part 2

To Read More

Modern React Redux Toolkit - Login ...

User Authentication is one of the common workflow in web applications. In this t...

Building Nodejs Microservice - A Cl...

This Article explains everything about how to build Nodejs Microservices in clou...

I Accidentally wiped the entire dat...

One of the tragic accident in my job turned out to be good learning for me in re...