EDUCBA Logo

EDUCBA

MENUMENU
  • Explore
    • EDUCBA Pro
    • PRO Bundles
    • Featured Skills
    • New & Trending
    • Fresh Entries
    • Finance
    • Data Science
    • Programming and Dev
    • Excel
    • Marketing
    • HR
    • PDP
    • VFX and Design
    • Project Management
    • Exam Prep
    • All Courses
  • Blog
  • Enterprise
  • Free Courses
  • Log in
  • Sign Up
Home Data Science Data Science Tutorials Kafka Tutorial Kafka offset
 

Kafka offset

Updated March 14, 2023

Kafka offset

 

 

Introduction to Kafka offset

In Kafka, the offset is a simple integer value. The same integer value will use by Kafka to maintain the current position of the consumer. Therefore, the offset plays a very important role while consuming the Kafka data. There are two types of offset, i.e., the current offset and the committed offset. If we do not need the duplicate copy data on the consumer front, then Kafka offset plays an important role. On the other hand, the committed offset means that the consumer has confirmed the processing position. Here, the processing term may vary from the Kafka architecture or project requirement.

Watch our Demo Courses and Videos

Valuation, Hadoop, Excel, Mobile Apps, Web Development & many more.

Syntax of the Kafka Offset

As such, there is no specific syntax available for the Kafka Offset. Generally, we are using the Kafka Offset value for the data consumption front on the consumer level.

Note: While working with the Kafka Offset. We are using the core Kafka commands and Kafka Offset terminology for the troubleshooting front.

How Kafka Offset Works?

The Kafka offset is majorly deal with in two different types, like the current offset and the committed offset. It will also be further divided into different parts also. Kafka is using the current offset to know the position of the Kafka consumer. While doing the partition rebalancing, the committed offset plays an important role.

Below is the property list and their value that we can use in the Kafka Offset.

  • flush.offset.checkpoint.interval.ms: It will help set up the persistent record frequency. The last flush instance will act as a log recovery point in the Kafka offset.
    Type: int
    Default: 60000 (1 minute)
    Valid Values: [0,…] Importance: high
    Update Mode: read-only
  • flush.scheduler.interval.ms: It will help to set up the frequency ms. The log flusher will check if any log needs to be flushed to disk level or not.
    Type: long
    Default: 9223372036854775807
    Valid Values:
    Importance: high
    Update Mode: read-only
  • flush.start.offset.checkpoint.interval.ms: It will help set up the frequency so that the persistent data or record of log start Kafka offset.
    Type: int
    Default: 60000 (1 minute)
    Valid Values:        [0,…] Importance: high
    Update Mode: read-only
  • metadata.max.bytes: The value is associated with the Kafka offset commit. It will deal with the maximum size for metadata.
    Type: int
    Default: 4096
    Valid Values:
    Importance: high
    Update Mode: read-only
  • commit.required.acks: Before doing any commit, it will require an acknowledgment. By default, the -1 value should not be overwriting.
    Type: short
    Default: -1
    Valid Values:
    Importance: high
    Update Mode: read-only
  • commit.timeout.ms: The Kafka offset commit will be running slow or delayed until all the running replicas for the offsets topic receive the final commit. In the second part, we can say that the timeout is reached. It will also be similar to the producer request timeout.
    Type: int
    Default: 5000 (5 seconds)
    Valid Values: [1,…] Importance: high
    Update Mode: read-only
  • load.buffer.size: It will help to define the batch size for reading operation from the offsets segments. It will load the offset into the volatile storage. It will deal with the soft limit. It will be overridden if records are too large and continually coming at high frequency.
    Type: int
    Default: 5242880
    Valid Values: [1,…] Importance: high
    Update Mode: read-only
  • retention.check.interval.ms: The frequency at which to check for stale offsets
    Type: long
    Default: 600000 (10 minutes)
    Valid Values: [1,…] Importance: high
    Update Mode: read-only
  • retention.minutes: When the consumer group lost all the consumers. In other words, we can say that it will become empty. Before getting discarded, the offsets.retention.minutes value will help to keep the offsets. The retention period will be applicable for standalone consumers. With the help of the last commit and the retention period, the offset will expire.
    Type: int
    Default: 10080
    Valid Values: [1,…] Importance: high
    Update Mode: read-only
  • topic.compression.codec: It will help to achieve the achieve “atomic” commits. The value help of the compression codec for the Kafka offsets topic.
    Type: int
    Default: 0
    Valid Values:
    Importance: high
    Update Mode: read-only
  • topic.num.partitions: It will help to define the number of partitions for the offset commit topic. Please make sure that it will not change after deployment.
    Type: int
    Default: 50
    Valid Values:        [1,…] Importance: high
    Update Mode: read-only
  • topic.replication.factor: It will help to define the replication factor of the Kafka offsets topic. If we keep the higher value, then it will have a higher entirety of the data.
    Type: short
    Default: 3
    Valid Values:        [1,…] Importance: high
    Update Mode: read-only
  • topic.segment.bytes: To do facilitate faster log compaction, we need to set less value. It will help for the faster log compaction and quick cache loads.
    Type: int
    Default: 104857600
    Valid Values:        [1,…] Importance: high
    Update Mode: read-only
  • interval.bytes: We need to keep the index value larger. If we keep the more indexing value, it will jump closer to the exact position.
    Type: int
    Default: 4096 (4 kibibytes)
    Valid Values:        [0,…] Server Default Property: log.index.interval.bytes
    Importance: medium
  • offset.reset: This configuration property help to define when there is no initial offset in Kafka.
    earliest: It will automatically reset the earliest offset
    latest: It will automatically reset the latest offset
    none: if no previous offset was found, it would throw a consumer exception
    anything else: throw the exception to the consumer.
    Type: string
    Default: latest
    Valid Values: [latest, earliest, none] Importance: medium
  • auto.commit: If we will configure the enable.auto.commit value is true; then the consumer offset will be committed in the background (The operation will be periodic in nature).
    Type: Boolean
    Default: true
    Valid Values:
    Importance: medium
  • flush.interval.ms: This configuration helps to try committing offsets for the tasks.
    Type: long
    Default: 60000 (1 minute)
    Valid Values:
    Importance: low
  • storage.partitions: When creating the offset for the storage topic. It will help to hold the number of partitions used.
    Type: int
    Default: 25
    Valid Values: Positive number or -1 to use the broker’s default
    Importance: low
  • storage.replication.factor: It will hold the replication information when we are creating the offset storage topic
    Type: short
    Default: 3
    Valid Values: Positive number not larger than the number of brokers in the Kafka cluster, or -1 to use the broker’s default
    Importance: low

Conclusion

We have seen the uncut concept of “Kafka Offset.” The offset is very important in terms of the data consumption front. Therefore, it will be very important to keep the offset value correct. If it will miss the match, then the data state will be inconsistent.

Recommended Articles

This is a guide to Kafka offset. Here we discuss the list of property and their value that we can use in the Kafka Offset and How it works. You may also have a look at the following articles to learn more –

  1. Kafka Monitoring
  2. Kafka Node
  3. Kafka MirrorMaker
  4. Kafka Tools

Primary Sidebar

Footer

Follow us!
  • EDUCBA FacebookEDUCBA TwitterEDUCBA LinkedINEDUCBA Instagram
  • EDUCBA YoutubeEDUCBA CourseraEDUCBA Udemy
APPS
EDUCBA Android AppEDUCBA iOS App
Blog
  • Blog
  • Free Tutorials
  • About us
  • Contact us
  • Log in
Courses
  • Enterprise Solutions
  • Free Courses
  • Explore Programs
  • All Courses
  • All in One Bundles
  • Sign up
Email
  • [email protected]

ISO 10004:2018 & ISO 9001:2015 Certified

© 2025 - EDUCBA. ALL RIGHTS RESERVED. THE CERTIFICATION NAMES ARE THE TRADEMARKS OF THEIR RESPECTIVE OWNERS.

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you
Loading . . .
Quiz
Question:

Answer:

Quiz Result
Total QuestionsCorrect AnswersWrong AnswersPercentage

Explore 1000+ varieties of Mock tests View more

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you
EDUCBA
Free Data Science Course

Hadoop, Data Science, Statistics & others

By continuing above step, you agree to our Terms of Use and Privacy Policy.
*Please provide your correct email id. Login details for this Free course will be emailed to you
EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you

EDUCBA Login

Forgot Password?

🚀 Limited Time Offer! - 🎁 ENROLL NOW