kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taylor Gautier <tgaut...@tagged.com>
Subject Re: Question - large number of topics
Date Fri, 19 Aug 2011 18:14:58 GMT
Thanks for the responses.

Coming back to this topic - on the wire protocol is it possible to register
interest for more than one topic - or is it 1:1 tcp connection to topic?

Inspecting the binary formats it looks like it has to be 1:1.

Thanks.

On Fri, Jul 22, 2011 at 4:37 PM, Jay Kreps <jay.kreps@gmail.com> wrote:

> Hi Taylor,
>
> I think you are correct the single-node scalability for the number of
> topics
> is not that great due to having multiple files per topic. I think the large
> directory problem can probably be mitigated by using a more modern
> filesystem, but as you and Jun point out ZK may also be strained.
>
> One thing that may not be obvious is it is not required to keep all topics
> on all machines, this will help scale the non-zk aspects. To do this you
> can
> either pre-create the topics or else add a custom partitioner which maps
> particular topics only to a subset of machines. In this way if you had, say
> 15 machines you could spread each topic over 3 machines and get 5X the max
> number of topics.
>
> -Jay
>
> On Fri, Jul 22, 2011 at 2:06 PM, Jun Rao <junrao@gmail.com> wrote:
>
> > Hi, Tayler,
> >
> > That's a good question. As your pointed out, a large number of topics
> will
> > put stress on local file directory and ZK. Maybe you can do a bit testing
> > first to see what breaks with a large number of topics. After that, we
> can
> > look into what needs to be fixed. Making the directory structure
> > hierarchical is a possibility.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Jul 22, 2011 at 1:23 PM, Taylor Gautier <tgautier@tagged.com>
> > wrote:
> >
> > > Hi.
> > >
> > > I am thinking to use kafka to send/receive messages for a large number
> of
> > > topics - order of 100k - 1M.
> > >
> > > It seems that the directory structure used for topics will probably not
> > > work
> > > for this usage.  Also, I'm not sure if the in-memory data structures
> > might
> > > suffer - and also it may be problematic for zookeeper.
> > >
> > > One thought I have is to modify the directory structure to be a tree of
> > > directories.  Not sure what if anything might need to be done to
> > in-memory
> > > structures or zookeeper info.
> > >
> > > Any thoughts?
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message