Kafka Console Consumer

Introduction to Kafka Console Consumer

The Kafka offers a command utility to view messages from the command line. Kafka offers the utility kafka-console-consumer.sh which helps to read messages from topic on command line. Kafka-console-consumer simply reads from a Kafka topic and writes data to console(standard output). By default, it outputs the raw bytes in the message with no formatting (using the Default Formatter).

How to Consume Data from Kafka using Kafka Console Consumer?

Command-line Argument

Following are the steps need to be followed for consuming messages from the topic

1. Starting Zookeeper

Kafka needs Zookeeper, it cannot start without Zookeeper. Start Zookeeper using the command zkServer.sh.

$ bin/zkServer.sh start

On your standard output you will see as follows:

Zookeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

Use the following command to connect to your local Zookeeper server:

bin/zkCli.sh -server 127.0.0.1:2181

2. Starting Kafka Server

To start the kafka broker service use the following command.

$  bin/kafka-server-start.sh-daemon
$ config/server.properties

After the Kafka broker is enabled, we can check that it functions by doing a few simple operations against the cluster creating a test subject, generating some messages and consuming the same messages.

3. Create Topic

Before we can consume messages from the topic, we first need to create a kafka topic, and to do so,we will use the utility that kafka provides to work on topics called kafka-topics.sh.

Let’s create a topic called “myTopic” with a single partition and a single replica:

$bin/kafka-topics.sh --create --zookeeper localhost:2181 -- replication-factor 1 --partitions 1 --topic myTopic

The replication-factor determines how many copies of the data will be produced. As we’re working with a single instance, keep this value at 1. Set the partitions as the amount of brokers you want your data to be divided between. As we’re running with a single broker, keep this value at 1.

4. Sending Messages to Kafka

Producer publishes message to one or more Kafka topics. It is responsible for putting data in out Kafka.We will use the utility that kafka provides to send messages to a topic using command line. The utility is called kafka-console- producer.sh. When you type any message in the terminal window it goes directly to that topic that specified while sending message. Bydefault, Kafka finds separate message as each line. Let’s start the producer then type in a couple of console messages to deliver to the server.

$ bin/kafka-console-producer.sh --broker-list localhost:9092 -- topic myTopic

• Welcome to KafkaConsole
• This is myTopic

You can either exit this command or have this terminal run for more testing. Now open the Kafka consumer process to a new terminal on the next step.

5. Using Kafka Console Consumer

Consumers connect to different topics, and read messages from brokers. They read data in consumer groups. Kafka provides a utility to read messages from topics by subscribing to it the utility is called kafka-console-consumer.sh.

Let’s run the consumer and consume all messages which were sent by the previous producer.

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 -
-topic myTopic --from-beginning Welcome to Kafka Console
This is my Topic

Console

The above steps are shown in console as follows:

1. Starting Zookeeper

Now let’s start the Zookeeper by running following command.

$ bin / zkServer.sh start
Zookeeper JMX enabled by default
Using config: /opt/zookeeper/bin/../conf/zo .cfg
Starting zookeeper ... STARTED

Run the following command to connect the local Zookeeper server. 

$ bin/zkCli.sh -server 127.0.0.1:2181

You may get a response with the CONNECTED name. It means you have a good installation of the local standalone Zookeeper.

Connecting to 127.0.0.1:2181
...
...
[zk: 127.0.0.1:2181(CONNECTED) 0]

2. Starting Kafka Server

Now go to your directory of Apache Kafka and execute following command.

$ bin/kafka-server-start.sh -daemon config/server.properties
[2020-02-10 10:47:45,989] INFO Kafka version: 1.0.1
(org.apache.kafka.common.utils.AppInfoParser)
[2020-02-10 10:47:45,995] INFO Kafka commitId: c0518aa65f25317e
(org.apache.kafka.common.utils.AppInfoParser)
[2020-02-10 10:47:46,006] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

Your Apache Kafka is up and running.

3. Creating Kafka Topic

To build a topic in the Kafka cluster, Kafka includes a file, kafka-topics.sh in the < KAFKA HOME>/bin / directory.

Let’s create topic with 6 partitions and 3 replication factor with topic name as myTopic. Running the script produces a topic called myTopic with 3 replicates and 6 partitions preserving metadata live in the Zookeeper:2181

$bin/kafka-topics.sh --create --zookeeper localhost:2181 -- replication-factor 1 --partitions 1 --topic myTopic

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 -- replication-factor 1 --partitions 1 --topic myTopic
Created topic "myTopic".

You will see the following output:

$ bin/kafka-topics.sh --list --zookeeper localhost:2181
myTopic

Now you can see the topic generated on kafka by running the list topic command.

Instead, you can also customize the brokers to auto-create topics when a non- existent topic is released to, instead of generating topics manually.

4. Sending Messages to Kafka Topic

Let’s send messages to kafka topic by starting producer using kafka-console- producer.shutility. Run the following command to launch a Kafka producer use console interface to write in the above sample topic created.

$ bin/kafka-console-producer.sh --broker-list localhost:9092 -- topic testTopic
Welcome to kafka This is my topic

5. Using Kafka Console Consumer

Kafka provides a utility to read messages from topics by subscribing to it the utility is called kafka-console-consumer.sh.

Let’s run the consumer and consume all messages which were sent by the previous producer.

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 -- topic testTopic --from-beginning
Welcome to kafka This is my topic

In a few cases,the kafka-console-consumer method can be helpful by having as an autonomous user of particular topics. This can be useful in comparing the results to a consumer system you’ve created.

Conclusion

So far, we have learned how to use kafka console consumer and how to create kafka topics and send messages using kafka console producer and read it on consumer terminals. Through operating the four components(zoo keeper, broker, producer and consumer in separate terminals, you will insert messages from the producer’s terminal and see them displayed in the terminal of the subscribing consumer.

This is a guide to Kafka Console Consumer . Here we discuss the introduction and how consumers read or consume data from kafka topics using kafka console consumer? You may also have a look at the following articles to learn more –

1. Kafka Tools
2. Kafka Architecture
3. Kafka Consumer Group
4. Kafka Applications
5. Kafka Node | 6 Best Steps
6. Kafka Replication | How to Work?