kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: High level consumer Question
Date Thu, 14 Jun 2012 00:09:38 GMT
Yes.

-Jay

On Wed, Jun 13, 2012 at 4:35 PM, Sam William <sampd@stumbleupon.com> wrote:

> So lets say I  have a topic with 100 partitions.  And Im running a
> high-kevel consumer with 25 threads. If I start a new consumer process with
> the same groupid and  say this has another 10 threads.  I can expect some
> partitions to be reassigned to the new consumers  right ?
>
> On Jun 11, 2012, at 8:05 PM, Sam William wrote:
>
> > Thanks Jay.. I had made such an assumption and seems like it was valid.
> >
> > Sam
> > On Jun 11, 2012, at 7:52 PM, Jay Kreps wrote:
> >
> >> Hi Sam,
> >>
> >> We assign partitions to consumer threads so that each partition has a
> >> consumer. If you have more partitions then threads, some threads will
> get
> >> multiple partitions (nothing wrong with this); if you have more threads
> >> than partitions some thread will not get any partitions and will be
> idle.
> >> But never will one partition go to more than one thread (since the
> >> consumption is meant to be ordered within that partition, splitting it
> >> would likely break this).
> >>
> >> -Jay
> >>
> >> On Mon, Jun 11, 2012 at 7:11 PM, Sam William <sampd@stumbleupon.com>
> wrote:
> >>
> >>> Hi,
> >>> Im trying out the high level consumer using the  ConsumerConnector API
> .
> >>> I have a topic with 100 partitions  .  What does it mean if I create
> >>> messagestreams with
> >>>
> >>>
> >>> consumerConnector.createMessageStreams(Predef.Map(topic -> partitions),
> >>> decoder)
> >>>
> >>> where partitions < 100  (say 50).  does the consumer read messages only
> >>> from 50 of the 100 partitions ?
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Sam William
> >>> sampd@stumbleupon.com
> >>>
> >>>
> >>>
> >>>
> >
> > Sam William
> > sampd@stumbleupon.com
> >
> >
> >
>
> Sam William
> sampd@stumbleupon.com
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message