kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: New consumer - poll/seek javadoc confusing, need clarification
Date Tue, 21 Jul 2015 16:15:44 GMT
On Tue, Jul 21, 2015 at 2:38 AM, Stevo Slavić <sslavic@gmail.com> wrote:

> Hello Apache Kafka community,
>
> I find new consumer poll/seek javadoc a bit confusing. Just by reading docs
> I'm not sure what the outcome will be, what is expected in following
> scenario:
>
> - kafkaConsumer is instantiated with auto-commit off
> - kafkaConsumer.subscribe(someTopic)
> - kafkaConsumer.position is called for every TopicPartition HLC is actively
> subscribed on
>
> and then when doing multiple poll calls in succession (without calling
> commit), does seek have to be called in between poll calls to position HLC
> to skip what was read in previous poll, or does HLC keep that state
> (position after poll) in memory, so that next poll (without seek in between
> two poll calls) will continue from where last poll stopped?
>

The position is tracked in-memory within the consumer, so as long as there
isn't a consumer rebalance, consumption will just proceed with subsequent
messages (i.e. the behavior I think most people would find intuitive).
However, if a rebalance occurs (another consumer instance joins the group
or some leave), then a partition may be assigned to an different consumer
instance that has no idea about the current position and will restart based
on the offset reset setting (because attempting to fetch the committed
offset will fail since no offsets have been committed).

-Ewen


> Could be it's just me not understanding this from javadoc. If not, maybe
> javadoc can be improved to make this (even) more obvious.
>
> Kind regards,
> Stevo Slavic.
>



-- 
Thanks,
Ewen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message