kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niclas Hedhman <nic...@hedhman.org>
Subject Re: Adding partitions...
Date Fri, 23 May 2014 12:51:59 GMT
Right... We don't yet know if there is going to be "meaning" to the
partition (such as content affinity of consumers has been discussed), I am
simply setting the stage of how things really work.


Thanks
Niclas


On Fri, May 23, 2014 at 2:41 PM, svante karlsson <saka@csi.se> wrote:

> No reshuffeling will take place. And reading messages and put them back in
> again will not remove the messages from their "old" partition so the same
> message will the exist in more than one partition - eventually to get aged
> out of the oldest partion.
>
> If you use partitioning to distribute the load then it's possible to just
> leave the old data in the existing partitions and let newer data fill all
> the partitions.
>
> If the partitioning means something to your applications then you can
> always create a new topic and populate that from the old topic. This will
> double your data for some time before you can age out the old topic.
>
>
> 2014-05-23 12:44 GMT+02:00 Niclas Hedhman <niclas@hedhman.org>:
>
> > Hi,
> >
> > we are trying to figure out how to layout our topics and partitions. And
> > one thing I can't find in the documentation is;
> >
> > What happens to data that is sitting in the 'old partitions' when I add a
> > new one?
> >
> > My gut feeling says that everything remains in the partitions as they
> are,
> > and if we need to have partitions associated with keys, we have to
> > implement this re-shuffling ourselves, i.e. read all the messages out and
> > put them back in.
> >
> > Is it so? And is there any hooks to assist in this?
> >
> >
> > Cheers
> > --
> > Niclas Hedhman, Software Developer
> > 河南南路555弄15号1901室。
> > http://www.qi4j.org - New Energy for Java
> >
> > I live here; http://tinyurl.com/3xugrbk
> > I work here; http://tinyurl.com/6a2pl4j
> > I relax here; http://tinyurl.com/2cgsug
> >
>



-- 
Niclas Hedhman, Software Developer
河南南路555弄15号1901室。
http://www.qi4j.org - New Energy for Java

I live here; http://tinyurl.com/3xugrbk
I work here; http://tinyurl.com/6a2pl4j
I relax here; http://tinyurl.com/2cgsug

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message