EDUCBA

EDUCBA

MENUMENU
  • Free Tutorials
  • Free Courses
  • Certification Courses
  • 360+ Courses All in One Bundle
  • Login
Home Data Science Data Science Tutorials Kafka Tutorial Kafka Node
Secondary Sidebar
Kafka Tutorial
  • Basic
    • What is Kafka?
    • Kafka Applications
    • Kafka Version
    • Kafka Use Cases
    • Kafka Consumer Group
    • Kafka Tools
    • Kafka Architecture
    • Kafka MirrorMaker
    • Kafka Console Producer
    • Kafka Console Consumer
    • Kafka Node
    • Kafka Listener
    • Kafka Cluster
    • Kafka Partition
    • Kafka Event
    • Kafka Replication
    • Kafka Monitoring
    • Kafka Zookeeper
    • Kafka Connect
    • Kafka Partition Key
    • Kafka Topic
    • Kafka burrow
    • Kafka Delete Topic
    • Kafka Replication Factor
    • Kafka Interview Questions
    • Kafka Alternatives
    • Kafka Queue
    • Kafka message
    • Kafka offset
    • Kafka Manager
    • Kafka Rebalance
    • Kafka Port
    • Kafka JDBC Connector
    • Kafka Security
    • Kafka confluent
    • Kafka Consumer
    • Kafka Producer
    • Kafka Client
    • Kafka Producer Config
    • Kafka Exporter
    • Kafka WebSocket
    • Kafka REST Proxy
    • Kafka Protocol
    • Kafka Producer Example

Related Courses

All in One Data Science Course

Pig Certification Course

Scala Certification Course

Kafka Node

By Priya PedamkarPriya Pedamkar

Kafka Node

Introduction to Kafka Node

Throughout the years, Kafka has evolved tremendously in many ways. Kafka Node is nothing but a Node.js client for Apache Kafka versions of 0.9 and later. We will learn more about the Kafka Node and its examples further.

Let us learn more about Kafka Node:

Start Your Free Data Science Course

Hadoop, Data Science, Statistics & others

Just to revise, Kafka is a distributed, partitioned and replicated pub-sub messaging system. Kafka stores messages in topics (partitioned and replicated across multiple brokers). Producers send messages to topics from which consumers or their consumer groups read. It is important, to begin with, the understanding of microservices before we address how Kafka-node operates. We are mainly in the Microservices Era of growth. Many businesses favour the use of Microservices to develop their applications.

The software architecture of microservices is an important model for software engineering and development with that frequency. As the size of applications increases, i.e. data consumed, processed, and generated, finding fault-tolerant, reliable ways of managing both the systems and the handled data becomes increasingly important. As the name suggests, a microservice is software that interacts with other system components to perform a function. Many use cases are event sources for microservices. We will implement Kafka for Event Sourcing using Node.js client.

All in One Data Science Bundle(360+ Courses, 50+ projects)
Python TutorialMachine LearningAWSArtificial Intelligence
TableauR ProgrammingPowerBIDeep Learning
Price
View Courses
360+ Online Courses | 50+ projects | 1500+ Hours | Verifiable Certificates | Lifetime Access
4.7 (86,112 ratings)

This is one of the most common cases in which a Microservice through Kafka talks with another Microservice.

Integrating Kafka with NodeJS

Let’s build a NodeJS API that is going to serve as a Kafka producer. Then we will create another consumer in NodeJS who will consume the topic that we are going to create first. Let’s follow the steps needed to consume messages from the topic using the command line.

1. Starting Zookeeper

It doesn’t begin without a Zookeeper; Kafka needs a zookeeper. Start Zookeeper by using the command zkServer.sh.

Command:

$bin/zkServer.sh start

Output:

Kafta Node Example

2. Start Kafka Server

Use the below command to start the Kafka server:

Command:

$ bin/kafka-server-start.sh -daemon $ config/server.properties

3. Topic Creation

Let’s create a topic called “example” with a single partition and a single replica:

Command:

$bin/kafka-topics.sh --create --zookeeper localhost:2181 -- replication-factor 1 --partitions 1 --topic example

The replication-factor determines how many copies of the data will be produced. As we’re working with a single instance, keep this value at 1. Set the partitions as the number of brokers you want your data to be divided between. As we’re running with a single broker, keep this value at 1.

4. Integrating Kafka with NodeJS

The next step is creating individual producers and consumers using Node.js. But before that, let’s connect our Kafka with NodeJS.

npm install kafka-node –save

Built the following config file:

module.exports = {'example',kafka_server: 'localhost:2181',};

5. Creating Producer

We need a producer to instantiate to publish messages with Kafka-node. A message on the given topic is published using the following function.

Code:

const kafka = require (‘kafka-node’);
const bp = require('body-parser');
const config = require('./config');
try {
const Producer = kafka.Producer;
const client = new kafka.Client(config.kafka_server);
const producer = new Producer(client);
const kafka_topic = 'example';
console.log(kafka_topic);
let payloads = [
{
topic: kafka_topic,
messages: config.kafka_topic
}
];
producer.on('ready', async function() {
let push_status = producer.send(payloads, (err, data) => {
if (err) {
console.log('[kafka-producer -> '+kafka_topic+']: broker failed');
} else {
console.log('[kafka-producer -> '+kafka_topic+']: broker success');
}
});
});
producer.on('error', function(err) {
console.log(err);
console.log('[kafka-producer -> '+kafka_topic+']: connection error');
throw err;
});
}
catch(e) {
console.log(e);
}

6. Creating Consumer

We will instantiate a customer to listen to messages on a specific topic. The following functionality consumes from a topic.

Code:

const kafka = require (‘kafka-node’);
const bp = require('body-parser');
const config = require('./config');
try {
const Consumer = kafka.HighLevelConsumer;
const client = new kafka.Client(config.kafka_server);
let consumer = new Consumer(
client,
[{ topic: config.kafka_topic, partition: 0 }],
{
autoCommit: true,
fetchMaxWaitMs: 1000,
fetchMaxBytes: 1024 * 1024,
encoding: 'utf8',
fromOffset: false
}
);
consumer.on('message', async function(message) {
console.log('here');
console.log(
'kafka-> ',
message.value
);
})
consumer.on('error', function(err) {
console.log('error', err);
});
}
catch(e) {
console.log(e);
}

Examples of Kafka Node

Let’s follow the below steps for creating a simple producer and consumer application in Node.js.

Kafka Node Example 1

1. Create a producer using the code below in the node shell while the ‘mytopics123’ topic is already created.

Producer.js

Kafka Node Example 2

2. Create a consumer using the code below in the node shell in the consumer.js file.

consumer.js

Kafka Node Example 3

All the available messages will be shown on the node terminal. Following is the output when we run the above consumer.js.

Running consumer.js:

Terminal Example 2

If you don’t create consumer.js and consume using Kafka-console-consumer.sh then you will get the following output:

Let’s run the Kafka-console-consumer.sh and consume all messages which the previous producer sent.

Running Kafka-console-consumer:

Console Consumer Example 2

Recommended Articles

This is a guide to Kafka Node. Here we discuss what the Kafka-node is and how to integrate the NodeJS with Kafka and its examples. You can also go through our other suggested articles to learn more –

  1. Kafka Alternatives | Top 5
  2. RabbitMQ vs Kafka – Top differences
  3. 10 Best Kafka Interview Questions
  4. A Quick Glance of Kafka Console Consumer
  5. Guide to Kafka Replication
Popular Course in this category
Apache Kafka Training (1 Course, 1 Project)
  1 Online Courses |  1 Hands-on Project |  7+ Hours |  Verifiable Certificate of Completion
4.5
Price

View Course

Related Courses

All in One Data Science Bundle (360+ Courses, 50+ projects)4.9
Apache Pig Training (2 Courses, 4+ Projects)4.8
Scala Programming Training (3 Courses,1Project)4.7
0 Shares
Share
Tweet
Share
Primary Sidebar
Footer
About Us
  • Blog
  • Who is EDUCBA?
  • Sign Up
  • Live Classes
  • Corporate Training
  • Certificate from Top Institutions
  • Contact Us
  • Verifiable Certificate
  • Reviews
  • Terms and Conditions
  • Privacy Policy
  •  
Apps
  • iPhone & iPad
  • Android
Resources
  • Free Courses
  • Database Management
  • Machine Learning
  • All Tutorials
Certification Courses
  • All Courses
  • Data Science Course - All in One Bundle
  • Machine Learning Course
  • Hadoop Certification Training
  • Cloud Computing Training Course
  • R Programming Course
  • AWS Training Course
  • SAS Training Course

ISO 10004:2018 & ISO 9001:2015 Certified

© 2022 - EDUCBA. ALL RIGHTS RESERVED. THE CERTIFICATION NAMES ARE THE TRADEMARKS OF THEIR RESPECTIVE OWNERS.

EDUCBA
Free Data Science Course

SPSS, Data visualization with Python, Matplotlib Library, Seaborn Package

*Please provide your correct email id. Login details for this Free course will be emailed to you

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA Login

Forgot Password?

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA
Free Data Science Course

Hadoop, Data Science, Statistics & others

*Please provide your correct email id. Login details for this Free course will be emailed to you

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you

By signing up, you agree to our Terms of Use and Privacy Policy.

Let’s Get Started

By signing up, you agree to our Terms of Use and Privacy Policy.

This website or its third-party tools use cookies, which are necessary to its functioning and required to achieve the purposes illustrated in the cookie policy. By closing this banner, scrolling this page, clicking a link or continuing to browse otherwise, you agree to our Privacy Policy

Loading . . .
Quiz
Question:

Answer:

Quiz Result
Total QuestionsCorrect AnswersWrong AnswersPercentage

Explore 1000+ varieties of Mock tests View more