EDUCBA Logo

EDUCBA

MENUMENU
  • Explore
    • EDUCBA Pro
    • PRO Bundles
    • Featured Skills
    • New & Trending
    • Fresh Entries
    • Finance
    • Data Science
    • Programming and Dev
    • Excel
    • Marketing
    • HR
    • PDP
    • VFX and Design
    • Project Management
    • Exam Prep
    • All Courses
  • Blog
  • Enterprise
  • Free Courses
  • Log in
  • Sign Up
Home Data Science Data Science Tutorials Kafka Tutorial Kafka Node
 

Kafka Node

Priya Pedamkar
Article byPriya Pedamkar

Updated February 28, 2023

Kafka Node

 

 

Introduction to Kafka Node

Throughout the years, Kafka has evolved tremendously in many ways. Kafka Node is nothing but a Node.js client for Apache Kafka versions of 0.9 and later. We will learn more about the Kafka Node and its examples further.

Watch our Demo Courses and Videos

Valuation, Hadoop, Excel, Mobile Apps, Web Development & many more.

Let us learn more about Kafka Node:

Just to revise, Kafka is a distributed, partitioned and replicated pub-sub messaging system. Kafka stores messages in topics (partitioned and replicated across multiple brokers). Producers send messages to topics from which consumers or their consumer groups read. It is important, to begin with, the understanding of microservices before we address how Kafka-node operates. We are mainly in the Microservices Era of growth. Many businesses favour the use of Microservices to develop their applications.

The software architecture of microservices is an important model for software engineering and development with that frequency. As the size of applications increases, i.e. data consumed, processed, and generated, finding fault-tolerant, reliable ways of managing both the systems and the handled data becomes increasingly important. As the name suggests, a microservice is software that interacts with other system components to perform a function. Many use cases are event sources for microservices. We will implement Kafka for Event Sourcing using Node.js client.

This is one of the most common cases in which a Microservice through Kafka talks with another Microservice.

Integrating Kafka with NodeJS

Let’s build a NodeJS API that is going to serve as a Kafka producer. Then we will create another consumer in NodeJS who will consume the topic that we are going to create first. Let’s follow the steps needed to consume messages from the topic using the command line.

1. Starting Zookeeper

It doesn’t begin without a Zookeeper; Kafka needs a zookeeper. Start Zookeeper by using the command zkServer.sh.

Command:

$bin/zkServer.sh start

Output:

Kafta Node Example

2. Start Kafka Server

Use the below command to start the Kafka server:

Command:

$ bin/kafka-server-start.sh -daemon $ config/server.properties

3. Topic Creation

Let’s create a topic called “example” with a single partition and a single replica:

Command:

$bin/kafka-topics.sh --create --zookeeper localhost:2181 -- replication-factor 1 --partitions 1 --topic example

The replication-factor determines how many copies of the data will be produced. As we’re working with a single instance, keep this value at 1. Set the partitions as the number of brokers you want your data to be divided between. As we’re running with a single broker, keep this value at 1.

4. Integrating Kafka with NodeJS

The next step is creating individual producers and consumers using Node.js. But before that, let’s connect our Kafka with NodeJS.

npm install kafka-node –save

Built the following config file:

module.exports = {'example',kafka_server: 'localhost:2181',};

5. Creating Producer

We need a producer to instantiate to publish messages with Kafka-node. A message on the given topic is published using the following function.

Code:

const kafka = require (‘kafka-node’);
const bp = require('body-parser');
const config = require('./config');
try {
const Producer = kafka.Producer;
const client = new kafka.Client(config.kafka_server);
const producer = new Producer(client);
const kafka_topic = 'example';
console.log(kafka_topic);
let payloads = [
{
topic: kafka_topic,
messages: config.kafka_topic
}
];
producer.on('ready', async function() {
let push_status = producer.send(payloads, (err, data) => {
if (err) {
console.log('[kafka-producer -> '+kafka_topic+']: broker failed');
} else {
console.log('[kafka-producer -> '+kafka_topic+']: broker success');
}
});
});
producer.on('error', function(err) {
console.log(err);
console.log('[kafka-producer -> '+kafka_topic+']: connection error');
throw err;
});
}
catch(e) {
console.log(e);
}

6. Creating Consumer

We will instantiate a customer to listen to messages on a specific topic. The following functionality consumes from a topic.

Code:

const kafka = require (‘kafka-node’);
const bp = require('body-parser');
const config = require('./config');
try {
const Consumer = kafka.HighLevelConsumer;
const client = new kafka.Client(config.kafka_server);
let consumer = new Consumer(
client,
[{ topic: config.kafka_topic, partition: 0 }],
{
autoCommit: true,
fetchMaxWaitMs: 1000,
fetchMaxBytes: 1024 * 1024,
encoding: 'utf8',
fromOffset: false
}
);
consumer.on('message', async function(message) {
console.log('here');
console.log(
'kafka-> ',
message.value
);
})
consumer.on('error', function(err) {
console.log('error', err);
});
}
catch(e) {
console.log(e);
}

Examples of Kafka Node

Let’s follow the below steps for creating a simple producer and consumer application in Node.js.

Kafka Node Example 1

1. Create a producer using the code below in the node shell while the ‘mytopics123’ topic is already created.

Producer.js

Kafka Node Example 2

2. Create a consumer using the code below in the node shell in the consumer.js file.

consumer.js

Kafka Node Example 3

All the available messages will be shown on the node terminal. Following is the output when we run the above consumer.js.

Running consumer.js:

Terminal Example 2

If you don’t create consumer.js and consume using Kafka-console-consumer.sh then you will get the following output:

Let’s run the Kafka-console-consumer.sh and consume all messages which the previous producer sent.

Running Kafka-console-consumer:

Console Consumer Example 2

Recommended Articles

This is a guide to Kafka Node. Here we discuss what the Kafka-node is and how to integrate the NodeJS with Kafka and its examples. You can also go through our other suggested articles to learn more –

  1. Kafka Alternatives | Top 5
  2. RabbitMQ vs Kafka – Top differences
  3. 10 Best Kafka Interview Questions
  4. A Quick Glance of Kafka Console Consumer
  5. Guide to Kafka Replication

Primary Sidebar

Footer

Follow us!
  • EDUCBA FacebookEDUCBA TwitterEDUCBA LinkedINEDUCBA Instagram
  • EDUCBA YoutubeEDUCBA CourseraEDUCBA Udemy
APPS
EDUCBA Android AppEDUCBA iOS App
Blog
  • Blog
  • Free Tutorials
  • About us
  • Contact us
  • Log in
Courses
  • Enterprise Solutions
  • Free Courses
  • Explore Programs
  • All Courses
  • All in One Bundles
  • Sign up
Email
  • [email protected]

ISO 10004:2018 & ISO 9001:2015 Certified

© 2025 - EDUCBA. ALL RIGHTS RESERVED. THE CERTIFICATION NAMES ARE THE TRADEMARKS OF THEIR RESPECTIVE OWNERS.

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you
Loading . . .
Quiz
Question:

Answer:

Quiz Result
Total QuestionsCorrect AnswersWrong AnswersPercentage

Explore 1000+ varieties of Mock tests View more

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you
EDUCBA
Free Data Science Course

Hadoop, Data Science, Statistics & others

By continuing above step, you agree to our Terms of Use and Privacy Policy.
*Please provide your correct email id. Login details for this Free course will be emailed to you
EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you

EDUCBA Login

Forgot Password?

🚀 Limited Time Offer! - 🎁 ENROLL NOW