kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: question about threaded vs. serial consumers
Date Tue, 10 Jul 2012 00:02:11 GMT
That is correct, you have exactly those options.

People do both single threaded multi-jvm consumers, multi-threaded
multi-jvm consumers, and single-threaded single jvm consumers. There is not
particular advantage to one or the other unless the processors need to
share data of some sort (in which case threads may be more convenient).

We do not have the equivalent of the unix poll/select api over partitions
or topics. That would be a good thing for us to add.

Load balancing is up to the producer (messages are partitioned based on a
key or randomly), so whether it is possible for a partition to not get any
events is somewhat situation dependent.

-Jay

On Mon, Jul 9, 2012 at 2:07 PM, James A. Robinson <jim.robinson@stanford.edu
> wrote:

> Hi folks,
>
> Reading through the Java examples for Kafka consumers, it appeared to
> me as though I have the following options when it comes to setting up
> consumers for a topic with more than one partition:
>
>   (a) Write a consumer that kicks off N threads, one thread per
>       partition, to read messages from the broker in parallel.
>
>   (b) Write a single threaded consumer and run N different JVMs, each
>       reading one of N partitions.
>
>   (c) Write a single threaded consumer that iterates through each
>       partition, using a consumer.timeout.ms setting to trigger a
>       timeout so that I don't block forever on a single partition that
>       is not being written to.
>
> Are there are options?  Is there a way to set up a non-blocking poll
> of a KafkaStream to see whether or not it has a message available?
>
> Does anyone actually do either (b) or (c), or is (a) really the only
> realistic option?  Is it possible/likely that a topic with N
> partitions might have one partition that is never updated?
>
> Jim
>
> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> James A. Robinson                       jim.robinson@stanford.edu
> Stanford University HighWire Press      http://highwire.stanford.edu/
> +1 650 7237294 (Work)                   +1 650 7259335 (Fax)
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message