Kafka Producer

Introduction to Kafka Producer

A Kafka producer is defined as; it can produce data on the topic with various separators, in which it can impulsively understand what data need to be put down to which separator and broker; logically, the Kafka producer is significantly more accessible than the consumer because it does not require of collaboration, and a separator can able to plan every message to the topic separator and producer can able to convey a producer request to the chief of that separator. Moreover, the separator assures Kafka that all messages with an equal non-empty key will not convey to an equivalent separator.

What is Kafka Producer?

The Kafka producer can produce or communicate the data to the topics inside the various separations. The producer can impulsively know what data should be communicated to which separation and which broker, and the user does not need to describe the broker and the partition. Instead, it can utilize the message keys and acknowledgment action plan to communicate the data to the cluster. The messages can authorize the idea of the key for sending the messages in a particular order in which the key can be authorized either by sending the data to every separation or by sending the data to a specific separation.

Kafka Producer Configuration

Let us see the configuration setting to check how they act on the bearing of the producer:

• Core configuration: In this configuration, we need to set the ‘bootstrap. Servers’ property so that the producer can search out the Kafka cluster; despite that, we always need to set a ‘client id’ as long as it can allow us to correspond the requests to the broker concerning the client.
• Message infidelity: We can able to manage the resilience of messages which are written to Kafka by setting the ‘acks,’ and the default value of ‘0’ can be accepted from the separator, and we can also be able to utilize the value of ‘0’ to promote the output, but in such case, we do not have the assurance that not only did the separation leader which can accept to write.
• Message ordering: Messages can be written to the broker in the identical order; they can accept by the producer client if we authorize them to send messages by setting the ‘retries’ to a value bigger than zero.

Kafka Producer API

The ‘KafkaProducer’ is the middle segment of the KafkaProducer API. It can give a choice to join a Kafka broker in its constructor using the ProducerRecord and Callback methods.

The identification of the send() method, which is provided by the KafkaProducer class, is given below:

Code:

"producer.send(new ProducerRecord<byte[],byte[]>(topic,
partition, key1, value1), callback);",

• ProducerRecord: It can support the buffer of records held for sending.
• Callback: The user can provide this method for implementation when the server has accepted the record.

The Kafka producer class has been issued a ‘flush’ method to ensure that all messages sent are completed literally.

• public void flush (): KafkaProducer class can have a separation method in which it can support the separation of metadata related to the given topic
• public map metrics (): It can give back internal metrics supported by the producer.
• public void close (): This method has been provided to build all previously sent requests are concluded.

Example of Kafka Producer

Let us see the step-by-step procedure to write a simple example of a producer in Apache Kafka with Java:

• To generate the Java project: At first, we have to generate a new Java project.
• Append jars to construct path: There are some jar files that we need to append to create a path for constructing a Java project; we can download the jars from the link ‘https://kafka.apache.org/downloads’

kafka_2.11-0.11.0.0.jar
kafka-clients-0.11.0.0.jar
scala-library-2.12.3.jar
slf4j-api-1.7.25.jar
slf4j-log4j12-1.7.25.jar
log4j-1.2.17.jar

• New TestProducer thread: After that, we have to generate a new class for test producer TestProducer.java which extends ‘Thread’ so that the producer can set in motion a new thread derives from the machine on request.

Code:

public class TestProducer extends Thread {
. . .
}

• Properties of kafka producer: To supply the information like Kafka server URL, Kafka server port, client ID, and the serializer for key and value.

• To generate the Kafka producer using properties: Using the properties described above, let us try to generate the new KafkaProducer.

Code:

“KafkaProducer producer = new KafkaProducer < > (properties);”

• Synchronous or asynchronous: We can dispatch the events from the producer to the server of Kafka simultaneously or non-synchronously.

We can convey messages simultaneously as defined below; when the messages are conveyed simultaneously, they are subject to the end of the communication to the Kafka server. The InterruptedException and the ExecutionException can manage the exception.

Code:

try {
  producer.send(new ProducerRecord<>(topic,
   messageNo,
   messageStr)).get();
  }
catch (InterruptedException | ExecutionException e)
  {
e.printStackTrace();
  }

When messages are dispatched non-simultaneously, we have to give a callback class that can execute the onCompletion() method that can be called when a message is dispatched effectively, and the Kafka server can accept that.

Code:

producer.send(new ProducerRecord<>(topic,
  messageNo,
  messageStr), new DemoCallBack(beginTime, messageNo, messageStr));

• To begin Zookeeper and Kafka cluster: For traversing to the origin of the Kafka directory and moving every command in discrete terminals to begin ZooKeeper and Kafka Cluster.

Code:

$ bin/zookeeper-server-start.sh config/zookeeper.properties
$ bin/kafka-server-start.sh config/server.properties

• To begin a SampleProducer thread:

Code:

SampleProducer producerThread = new TrialProducer(TOPIC, isAsync);
producerThread.start();

Conclusion

In this article, we conclude that the Kafka producer is responsible for sending the data to the various separators, which are more accessible than the consumer; we have also discussed the configuration, API, and examples related to the Kafka producer.

Recommended Articles

This is a guide to Kafka Producer. Here we discuss the introduction, Kafka producer configuration, and examples. You may also have a look at the following articles to learn more –

1. Kafka offset
2. Kafka message
3. Kafka Queue
4. Kafka Connect