kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nag Y <andriod.nag.u...@gmail.com>
Subject Kafka - Auto commit
Date Fri, 12 Jun 2020 12:17:18 GMT
During the time when i was going through the doc
<https://www.safaribooksonline.com/library/view/kafka-the-definitive/9781491936153/ch04.html#callout_kafka_consumers__reading_data_from_kafka_CO2-1>
and
come across ,

Automatic Commit The easiest way to commit offsets is to allow the consumer
to do it for you. If you configure enable.auto.commit=true, then every five
seconds the consumer will commit the largest offset your client received
from poll(). The five-second interval is the default and is controlled by
setting auto.commit.interval.ms. Just like everything else in the consumer,
the automatic commits are driven by the poll loop. Whenever you poll, the
consumer checks if it is time to commit, and if it is, it will commit the
offsets it returned in the last poll.

And also, How does kafka consumer auto commit work?
<https://stackoverflow.com/questions/46546489/how-does-kafka-consumer-auto-commit-work?rq=1>

The auto-commit check is called in every poll and it checks that the "time
elapsed is greater than the configured time". If so, the offset is
committed.

In case the commit interval is 5 seconds and poll is happening in 7
seconds, the commit will happen after 7 seconds only.

However, if we take a close look, the auto commit doesnt seem actually
happen every 5 sec ( or time interval configured through "
auto.commit.interval.ms ) but happens every time if "time elapsed" is
greater than "auto.commit.interval.ms" and intervals of "time elapsed"+"
auto.commit.interval.ms" -- which means it doesn't necessarily commit the
offset every interval, configured thorough "auto.commit.interval.ms".

Please add your thoughts

Update #1

It is adding up confusion after going through more details , Can someone
add more details about this - will poll() method happens in the background
at 5 sec which is different from poll() method issued from the consumer ?

The poll() call is issued in the background at the set
auto.commit.interval.ms.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message