kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dennis Haller <dhal...@talemetry.com>
Subject Re: How does Kafka decide which Consumer out of multiple Consumer clients to assign to a single topic partition
Date Thu, 22 Aug 2013 17:46:19 GMT
Thanks Guozhang. It's good to have this in the FAQ.



On Thu, Aug 22, 2013 at 10:22 AM, Guozhang Wang <wangguoz@gmail.com> wrote:

> Thanks Neha for the clarification. I have created a new entry in FAQ for
> this question:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-CanIpredicttheresultsoftheconsumerrebabalance%3F
>
> Dennis, please let me know if this does not fully answer your question.
>
> Guozhang
>
>
> On Thu, Aug 22, 2013 at 9:29 AM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > We range partition a sorted list of topic-partitions over a sorted list
> of
> > consumer instances. This makes the rebalancing algorithm deterministic.
> As
> > soon as you bring up the 2nd consumer instance, if its position in the
> > sorted list is before the position of the 1st consumer client, it will
> end
> > up owning the partition.
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Aug 22, 2013 at 8:53 AM, Dennis Haller <dhaller@talemetry.com
> > >wrote:
> >
> > > I have a situation where two high level consumers are being created to
> > > consume a single topic. There is only one partition for the topic, so I
> > > understand that only one Consumer will end up owning the topic and
> > > receiving messages. The two consumers are created from two servers in a
> > > redundant master - slave configuration, and it is our intention that
> the
> > > servers should start in the same configuration predictably, with all
> the
> > > Consumers active on the master server.
> > >
> > > However, we find after both Consumers have been created that sometimes
> > the
> > > first Consumer client succeeds in owning the topic and sometimes it is
> > the
> > > second. I first thought that the first Consumer  client to register
> with
> > > the topic would be retained even if subsequent Consumers also register
> > for
> > > the topic, but I see sometimes the second Consumer client succeeds in
> > > replacing the first Consumer.
> > >
> > > The logs show a rebalancing algorithm working after each Consumer is
> > > registered.
> > >
> > > In this case, where there is only one topic-partition, is it possible
> to
> > > predict what Consumer client will own the topic? How is that
> rebalancing
> > > done?
> > >
> > > Thanks
> > > Dennis
> > >
> >
>
>
>
> --
> -- Guozhang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message