kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Curtin <curtin.ch...@gmail.com>
Subject Re: Is there a limitation on the number of simultaneous consumer connections to the same topic
Date Mon, 13 May 2013 13:10:26 GMT
Yes. However be aware that starting and stopping processes will cause a
rebalance of the consumers, so your code may find itself receiving events
from a different partition suddenly (so don't assume the partition you are
reading isn't going to change!)  Also as things are starting up you may
find a process receives many partitions at first, but as the other
processes are started the partitions get reassigned.

Finally, running more processes than partitions will mean those other
processes are idle.




On Mon, May 13, 2013 at 8:49 AM, Ming Li <lm0909@gmail.com> wrote:

> Hi Andrea,
>
> Thanks for your reply~~ you mean, it is no difference between
> having N threads share the same ConsumerConnector created by
> Consumer.createJavaConsumerConnector,
> and
> having N consumer process which has its own ConsumerConnector in every one
> of them?
>
> Best Regards,
> Li Ming
>
>
>
> On Mon, May 13, 2013 at 6:43 PM, Andrea Gazzarini <
> andrea.gazzarini@gmail.com> wrote:
>
> > It shouldn't.
> > Creating several listener / consumer processes belonging to the same
> group
> > means you are working with a point-to-point message channel so incoming
> > messages will be delivered only to one consumer.
> >
> > Maybe I'm wrong but I believe in that scenario there's no difference
> (from
> > broker perspective) between threads and processes.
> >
> > Regards,
> > Andrea
> >
> >
> > On 05/13/2013 12:15 PM, Ming Li wrote:
> >
> >> Hi,
> >>
> >> Does Kafka have a limitation on the simultaneous connections (created
> with
> >> Consumer.**createJavaConsumerConnector) for the same topic within the
> >> same
> >> group?
> >>
> >> My scenario is I need to consume a topic from different process (not
> >> thread), so I need to create lots of high level consumers.
> >>
> >>
> >> Best Regards,
> >> Li Ming
> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message