kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Question - large number of topics
Date Fri, 22 Jul 2011 21:06:58 GMT
Hi, Tayler,

That's a good question. As your pointed out, a large number of topics will
put stress on local file directory and ZK. Maybe you can do a bit testing
first to see what breaks with a large number of topics. After that, we can
look into what needs to be fixed. Making the directory structure
hierarchical is a possibility.

Thanks,

Jun


On Fri, Jul 22, 2011 at 1:23 PM, Taylor Gautier <tgautier@tagged.com> wrote:

> Hi.
>
> I am thinking to use kafka to send/receive messages for a large number of
> topics - order of 100k - 1M.
>
> It seems that the directory structure used for topics will probably not
> work
> for this usage.  Also, I'm not sure if the in-memory data structures might
> suffer - and also it may be problematic for zookeeper.
>
> One thought I have is to modify the directory structure to be a tree of
> directories.  Not sure what if anything might need to be done to in-memory
> structures or zookeeper info.
>
> Any thoughts?
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message