kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Janardhanan V S <janardha...@exotel.in>
Subject Re: Rebalancing and Its Impact
Date Tue, 19 Mar 2019 10:34:48 GMT
Hey everyone,

Requesting some help here..

Thanks,
Janardhanan V S

On Sat, Mar 16, 2019 at 7:36 AM Janardhanan V S <janardhanan@exotel.in>
wrote:

> Hi,
>
> I'm new to Kafka and I'm trying to design a wrapper library in both Java
> and Go (uses Confluent/Kafka-Go) for Kafka to be used internally. For my
> use-case, CommitSync is a crucial step and we should do a read only after
> properly committing the old one. Repeated processing is not a big issue and
> our client service is idempotent enough. But data loss is a major issue and
> should not occur.
>
> I will create X number of consumers initially and will keep on polling
> from them. Hence I would like to know more about the negative scenario's
> that could happen here, Impact of them and how to properly handle them.
>
> I would like to know more about:
>
> 1) Network issue during consumer processing:
>      What happens when network goes of for a brief period and comes back?
> Does Kafka consumer automatically handle this and becomes alive when
> network comes back or do we have to reinitialise them? If they come back
> alive do they resume work from where they left of?
> Eg: Consumer X read 50 records from Partition Y. Now internally the
> consumer offset moved to +50. But before committing network issue happens
> and the comes back alive. Now will the consumer have the metadata about
> what it read for last poll. Can it go on to commit +50 in offset?
>
> 2) Rebalancing in consumer groups. Impact of them on existing consumer
> process - whether the existing working consumer instance will pause and
> resume work during a rebalance or do we have to reinitialize them? How long
> can rebalance occur? If the consumer comes back alive after rebalance, does
> it have metadata about it last read?
>
> 3) What happens when a consumer joins during a rebalancing. Ideally it is
> again a rebalancing scenario. What will happen now? The existing will be
> discarded and the new one starts or will wait for the existing rebalance to
> complete?
>
> Kindly help me understanding these scenarios is detail and suggest
> solutions if possible. Also it would be much more helpful, if you could
> point me to a resource - an online article / book or anything that provides
> insight into the intricate details of Kafka.
>
>
>
> Thanks and Regards,
> Janardhanan V S
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message