EDUCBA

EDUCBA

MENUMENU
  • Free Tutorials
  • Free Courses
  • Certification Courses
  • 360+ Courses All in One Bundle
  • Login
Home Data Science Data Science Tutorials Kafka Tutorial How to Install Kafka?
Secondary Sidebar
Installation of Software Tutorial
  • Installation of Software
    • Autodesk Uninstall Tool
    • How To Install Apache
    • How To Install Appium
    • How to Install CakePHP
    • How to Install CentOS
    • How to Install Drupal 8
    • How to Install Fedora
    • How to Install FreeBSD
    • How to Install Go
    • How to Install Groovy
    • How To Install Joomla
    • How to Install Kafka
    • How to Install Kubernetes
    • How to Install Raspberry Pi
    • How to Install Typescript
    • How To Install WiX
    • Install Jupyter Notebook
    • Install Kubernetes Dashboard
    • Install NLTK
    • Install OpenStack
    • Install WordPress
    • How to Install Splunk
    • How To Install SVN
    • Install Samba
    • WordPress Login

Related Courses

Python Certification Training

Java Training

HTML Training

How to Install Kafka?

By Aanchal SinghAanchal Singh

How to Install Kafka?

How to Install Kafka?

Kafka is a distributed streaming platform. It helps in publishing and subscribing to streams of records. Kafka is usually used for building real-time streaming data pipelines that reliably get data between different systems and applications. It can also help build real-time streaming applications that help transform or make changes with the streams of data. Kafka enables to store of streams of records in different categories called topics. Each record consists of a key, a value, and a timestamp. It has four major components like Producer API, Consumer API, Streams API, and Connector API. It uses the TCP protocol, which is good communication between clients and servers with high performance. In order to install Kafka, the following steps can be followed.

Install Kafka on Windows OS

For installing Kafka on Windows, follow the below steps:

Step 1: Download Zookeeper from the following link: HTTP: //zookeeper.apache.org/releases.html

Once this is done, download Kafka from following the link: https://kafka.apache.org/downloads.html.

Start Your Free Data Science Course

Hadoop, Data Science, Statistics & others

Also, download JRE as per your operating system and CPU architecture from the below link: https://www.oracle.com/java/technologies/downloads/#java8.

All in One Data Science Bundle(360+ Courses, 50+ projects)
Python TutorialMachine LearningAWSArtificial Intelligence
TableauR ProgrammingPowerBIDeep Learning
Price
View Courses
360+ Online Courses | 50+ projects | 1500+ Hours | Verifiable Certificates | Lifetime Access
4.7 (86,650 ratings)

Step 2: JDK Setup

Start JRE Installation and then click on the ‘Change Destination folder’ checkbox. Now, click on Install.

Once this is done, change the directory to any path without spaces in the folder name.

JDK Setup

Step 3: Once this is done open the environment variables from Control Panel -> System -> Advanced system settings -> Environment variables.

Step 4: Click on New User Variable in the User Variable tab and then type JAVA_HOME in the variable name. Enter your JRE path in the Variable value section.

It should be as below:

Set JRE path

Step 5: Now click on, OK.

Step 6: Search a path variable in the System Variable’ section in the “Environment Variable” dialogue box, which is opened.

Step 7: Edit the path and type “;%JAVA_HOME%\bin” at the end of the text already written there, just like the image below:

Kafka Installation 3

Step 8: To check if Java is installed properly, go to Command Prompt and type ‘java – version’. The version of Java that is installed will be displayed on the screen.

Kafka Installation 4

If you see the above details on the command prompt, then you are good from the Java side.

Once Java is installed, you can now move to get the setup of Zookeeper done.

Install Zookeeper:

Follow the below steps to get Zookeeper installed on your system:

  • Open the directory where the Zookeeper config directory is there. It can be C:\zookeeper-3.4.7\conf.
  • Now rename file “zoo_sample.cfg” to “cfg”
  • Open this renamed file in Notepad.
  • Find and edit the following: dataDir=/tmp/zookeeper to \zookeeper-3.4.7\data.
  • Just like we added an environment variable for Java, add an environment variable for Zookeeper.
  • Set the system variable path as: dataDir=/tmp/zookeeper to :\zookeeper-3.4.7\data
  • Edit the system variable named ‘Path’ and add; %ZOOKEEPER_HOME%\bin;
  • The default port for Zookeeper is 2181, which can be changed to any other port by going to the zoo.cfg file.
  • Run Zookeeper by opening a new cmd and type ‘zkserver’. You will see the below details.

Kafka Installation 5

Once you see this screen, you should be sure that Zookeeper has been installed successfully.

Setting up Kafka:

Once you have Java and Zookeeper up and running on your system, you can go ahead to set up Kafka.

  • Go to Kafka config directory.
  • Edit the file “server.properties”.
  • Once this is done, you can find and edit the line where you see: dirs=/tmp/Kafka-logs to “log.dir= C:\kafka_2.11-0.9.0.0\kafka-logs
  • If you have your Zookeeper running on some other machine, then you can change this path to “zookeeper.connect:2181” to a customized IP and port id.
  • The default port for Kafka is port 9092, and to connect to Zookeeper, it is 2181.

Running a Kafka Server:

Once the initial setup is done, you can easily run a Kafka server.

Before running the Kafka server, one must ensure that the Zookeeper instance is up and running.

1. Go to Kafka installation directory: C:/kafka_2.11-0.9.0.0

2. Open the command prompt, and press Shift+right click and choose the ‘Open command window here option.

3. Now type .\bin\windows\kafka-server-start.bat .\config\server.properties and press Enter.

Once you run this and everything is fine, then your screen should look as below:

Kafka Installation 6

4. Now, your Kafka server is up and running. You can create your own topics to store different messages. Once this is done, you can produce and consume Java or Scala code data or directly run from a command prompt.

Install Kafka on Linux

Follow the steps below to install Kafka on Linux:

Step 1: Download and extract Kafka binaries and store them in directories.

Step 2: Extract the archive you download using the tar command.

Step 3: To configure Kafka, go to server.properties. Open this file using the nano command and add the following at the bottom of the file.

Code:

nano ~/Kafka/config/server.properties

delete.topic.enable = true

Step 4: Once this is done, the user will have to create system unit files for Kafka services. This helps perform common service actions like starting up, stopping, and consistently restarting Kafka with other Linux services. Zookeeper is a service that Kafka uses in order to manage its cluster and configurations.

Step 5: To create a unit file for Zookeeper follows below:

Code:

sudo nano /etc/systemd/system/zookeeper.service

Step 6. Once this zookeeper file is created, paste below in it:

Code:

[Unit] Requires=network.target remote-fs.target
After=network.target remotefs.target
[Service] Type=simple
User=kafka
ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
[Install] WantedBy=multi-user.target

This [Unit] section here specifies that Zookeeper requires networking and filesystem to be ready before it can start.

The [Service] section lets the system know that zookeeper-server-start.sh and zookeeper-server-stop.sh files are present for starting and stopping the services.

Step 7. Now the user should create a system file for Kafka as below:

Code:

sudo nano /etc/system/system/Kafka.service

Step 8. In this file, paste below:

Code:

[Unit] Requires=zookeeper.service
After=zookeeper.service
[Service] Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh
/home/kafka/kafka/config/server.properties >
/home/kafka/kafka/kafka.log 2>&1'
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.shRestart=on-abnormal
[Install] WantedBy=multi-user.target

Here the [Unit] specifies that the unit file is dependent on zookeeper.service. This ensures that the zookeeper is started before Kafka starts.

Step 9: You now need to enable Kafka and reboot the server. Run: sudo systemctl enable Kafka.

Step 10: Testing Installation.

You can test your Kafka Installation by creating a topic and then publishing it to consumers.

Create a topic by: ~/Kafka/bin/Kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic TutorialTopic.

Post this; the user can create producers and consumers and then publish any data on specific topics.

Recommended Articles

This has been a guide to Install Kafka. Here we have discussed different steps to install Kafka on Windows and Linux. You may also look at the following articles to learn more –

  1. Step to Install Java 8
  2. How to Install C
  3. Guide To Install Ruby
  4. Kafka Applications
Popular Course in this category
Hadoop Training Program (20 Courses, 14+ Projects, 4 Quizzes)
  20 Online Courses |  14 Hands-on Projects |  135+ Hours |  Verifiable Certificate of Completion
4.5
Price

View Course

Related Courses

All in One Data Science Bundle (360+ Courses, 50+ projects)4.9
Apache Pig Training (2 Courses, 4+ Projects)4.8
Scala Programming Training (3 Courses,1Project)4.7
0 Shares
Share
Tweet
Share
Primary Sidebar
Footer
About Us
  • Blog
  • Who is EDUCBA?
  • Sign Up
  • Live Classes
  • Corporate Training
  • Certificate from Top Institutions
  • Contact Us
  • Verifiable Certificate
  • Reviews
  • Terms and Conditions
  • Privacy Policy
  •  
Apps
  • iPhone & iPad
  • Android
Resources
  • Free Courses
  • Database Management
  • Machine Learning
  • All Tutorials
Certification Courses
  • All Courses
  • Data Science Course - All in One Bundle
  • Machine Learning Course
  • Hadoop Certification Training
  • Cloud Computing Training Course
  • R Programming Course
  • AWS Training Course
  • SAS Training Course

ISO 10004:2018 & ISO 9001:2015 Certified

© 2022 - EDUCBA. ALL RIGHTS RESERVED. THE CERTIFICATION NAMES ARE THE TRADEMARKS OF THEIR RESPECTIVE OWNERS.

EDUCBA
Free Data Science Course

SPSS, Data visualization with Python, Matplotlib Library, Seaborn Package

*Please provide your correct email id. Login details for this Free course will be emailed to you

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA Login

Forgot Password?

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA
Free Data Science Course

Hadoop, Data Science, Statistics & others

*Please provide your correct email id. Login details for this Free course will be emailed to you

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you

By signing up, you agree to our Terms of Use and Privacy Policy.

Let’s Get Started

By signing up, you agree to our Terms of Use and Privacy Policy.

This website or its third-party tools use cookies, which are necessary to its functioning and required to achieve the purposes illustrated in the cookie policy. By closing this banner, scrolling this page, clicking a link or continuing to browse otherwise, you agree to our Privacy Policy

Loading . . .
Quiz
Question:

Answer:

Quiz Result
Total QuestionsCorrect AnswersWrong AnswersPercentage

Explore 1000+ varieties of Mock tests View more