kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bateman, Matt" <mabate...@ebay.com>
Subject RE: More Information On Partitioning
Date Thu, 11 Aug 2011 21:45:13 GMT
Hi Jun,

>From the paper it appears that what I'm looking for is scheduled as "future" work. I think
this paragraph illustrates the functionality we're looking for:

"At the lowest level this is supported by semantically partitioning messages on the join key
during publishing so that all messages sent with a particular key go to the same partition
and hence arrive at a single consumer process. This provides the foundation for processing
distributed streams across a cluster of consumer machines."

We need to guarantee that all messages in a single partition are delivered to the _same_ consumer
process.

Am I correct in understanding that this is not currently possible?

One way of achieving this might be to use multiple topics and publish messages to a particular
topic which is being consumed by only one consumer process. Am I missing something fundamental
or might this work?

Thanks for your help,

Matt

-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com] 
Sent: Sunday, August 07, 2011 5:30 PM
To: kafka-users@incubator.apache.org
Subject: Re: More Information On Partitioning

You can take a  look at the Kafka paper in NetDB ( http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf).
If you have more questions after reading, feel free to ask again.

Jun

On Fri, Aug 5, 2011 at 6:08 PM, Bateman, Matt <mabateman@ebay.com> wrote:

> Hi All,
>
> The documentation on partitioning is a little light on the website. Is 
> there somewhere I can get more detail as to how this exactly works? 
> I'm interested from the deployment, load balancing and intended use 
> patterns perspectives.
>
> I'm happy to read whatever docs I've missed if pointed in the right 
> direction.
>
> Thanks,
>
> Matt
>

Mime
View raw message