kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Milind Vaidya <kava...@gmail.com>
Subject Re: Consumer Group, relabancing and partition uniqueness
Date Wed, 29 Jun 2016 22:25:43 GMT

Thanks, I got your point.

The documentation as well as diagram showing the mechanism of consumer
group indicates that,the partitions are shared disjointly by consumers in a
You also stated above "Each of your consumer will receive message for its
allocated partition for that they subscribed."

e.g. P1......P5 are partitions and we have C1....C5 consumers belonging to
same group. So is it correct to assume that C2 will consume for P4(say) and
not from any other partition. Similarly Ck will consume from Pm where 1 >=
k, m <= 5. If no rebalancing happens, as in none of the consumers dies, how
long will this combination sustain ? or random rebalance may happen after a
while leading to C2 consuming from P3 as against P4 from which it was
originally consuming.

I have my logs for the consumer, which indicate that partitions associated
with a consumer change periodically.
Is there any mechanism by which I can make sure a consumer consumes from a
particular partition for sufficient amount of time which is configurable
provided none of the consumers goes down triggering rebalance.

On Wed, Jun 29, 2016 at 3:02 PM, Spico Florin <spicoflorin@gmail.com> wrote:

> Hi!
>   By default kafka uses internally a round robin partitioner that will send
> the messages to the right partition based on the message key. Each of your
> consumer will receive message for its allocated partition for that they
> subscribed.
>   In case of rebalance, if you add more consumers than the partitions then
> some of the consumers will not get any data. If one of the consumers dies,
> then the remained consumers will get messages from the partitions depending
> on their client id. Kafka internally uses the client id (lexicogarphic
> order) to allocate the partitions.
> I hope that this give you an overview of what happens and somehow answer to
> your questions.
> Regards,
> florin
> On Thu, Jun 30, 2016 at 12:36 AM, Milind Vaidya <kavadsa@gmail.com> wrote:
> > Hi
> >
> > Background :
> >
> > I am using a java based multithreaded kafka consumer.
> >
> > Two instances of  this consumer are running on 2 different machines i.e.
> > one consumer process per box, and  belong to same consumer group.
> >
> > Internally each process has 2 threads each.
> >
> > Both the consumer processes consume from same topic "rawlogs" which has 4
> > partitions.
> >
> > Problem :
> >
> > As per the documentation of consumer group "each message published to a
> > topic is delivered to one consumer instance within each subscribing
> > consumer
> > group" . But is there any mechanism by which a each consumer consumes
> from
> > disjoint set of partitions too ? or each message from whichever partition
> > it is, will be given randomly to one of the consumers ?
> >
> > In case of rebalance, the partitions may get shuffled among consumers but
> > then again they should get divided into 2 disjoint sets one for each
> > consumer.
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message