kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Inder Pall <inder.p...@gmail.com>
Subject Consumer re-balance behavior
Date Mon, 24 Oct 2011 08:28:43 GMT

need some clarity and confirmation on the following behavior.

1. I have a topic T spread across two brokers (B1, B2)running on different
machines, each having 2 partitions configured for T. Totally 4 partitions
(1-0, 1-1, 2-0, 2-1)
2. Consumer C1 is part of group g1 and is consuming from from B1, B2 for T
3. Add a new consumer C2 part of g1

This is triggering a re balance across C1 & C2 and eventually C1 gets 1-0,
1-1 and C2 gets 2-0, 2-1.
P.S. - B1, C1 are sharing the same machine, same is the case with B2,C2

both consumers are getting partitions which are hosted on the same boxes. Is
this a coincidence or an optimization w.r.t locality of data and will always
be applied.

More questions
1. When would you want to have multiple partitions of the same topic hosted
on the same broker. Is it that you have 2 partitions of T on B1 and 10 on B2
and on re balance C1 & C2 would get 6 each.
2.  As in the above use-case, C1 has 1-0 & 1-1 partitions of T and adding
messages to B1 results in the messages being spread across both the
partitions. Is this behavior round robin or based on segment file size/other
3. Is it possible to configure #partitons based on topic, if so how?

-- Inder

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message