kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: More Information On Partitioning
Date Thu, 11 Aug 2011 23:41:06 GMT
Matt,

Today, data from a partition is always consumed by a single consumer, unless
a rebalance is triggered. A rebalance only happens when there is a broker or
consumer change (e.g., going up/down).

In the near future when we have replication, a partition will never go away
in theory. So, rebalance only happens when there is a consumer change.

Jun

On Thu, Aug 11, 2011 at 2:45 PM, Bateman, Matt <mabateman@ebay.com> wrote:

> Hi Jun,
>
> From the paper it appears that what I'm looking for is scheduled as
> "future" work. I think this paragraph illustrates the functionality we're
> looking for:
>
> "At the lowest level this is supported by semantically partitioning
> messages on the join key during publishing so that all messages sent with a
> particular key go to the same partition and hence arrive at a single
> consumer process. This provides the foundation for processing distributed
> streams across a cluster of consumer machines."
>
> We need to guarantee that all messages in a single partition are delivered
> to the _same_ consumer process.
>
> Am I correct in understanding that this is not currently possible?
>
> One way of achieving this might be to use multiple topics and publish
> messages to a particular topic which is being consumed by only one consumer
> process. Am I missing something fundamental or might this work?
>
> Thanks for your help,
>
> Matt
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Sunday, August 07, 2011 5:30 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: More Information On Partitioning
>
> You can take a  look at the Kafka paper in NetDB (
> http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf
> ).
> If you have more questions after reading, feel free to ask again.
>
> Jun
>
> On Fri, Aug 5, 2011 at 6:08 PM, Bateman, Matt <mabateman@ebay.com> wrote:
>
> > Hi All,
> >
> > The documentation on partitioning is a little light on the website. Is
> > there somewhere I can get more detail as to how this exactly works?
> > I'm interested from the deployment, load balancing and intended use
> > patterns perspectives.
> >
> > I'm happy to read whatever docs I've missed if pointed in the right
> > direction.
> >
> > Thanks,
> >
> > Matt
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message