kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From István <lecc...@gmail.com>
Subject Re: How many partition can one single machine handle in Kafka?
Date Thu, 23 Oct 2014 17:38:58 GMT
RAID has nothing to do with the overall availability of your system, it is
just increasing the per node reliability.

Regards,
Istvan

On Wed, Oct 22, 2014 at 11:01 AM, Gwen Shapira <gshapira@cloudera.com>
wrote:

> RAID-10?
> Interesting choice for a system where the data is already replicated
> between nodes. Is it to avoid the cost of large replication over the
> network? how large are these disks?
>
> On Wed, Oct 22, 2014 at 10:00 AM, Todd Palino <tpalino@gmail.com> wrote:
> > In fact there are many more than 4000 open files. Many of our brokers run
> > with 28,000+ open files (regular file handles, not network connections).
> In
> > our case, we're beefing up the disk performance as much as we can by
> > running in a RAID-10 configuration with 14 disks.
> >
> > -Todd
> >
> > On Tue, Oct 21, 2014 at 7:58 PM, Xiaobin She <xiaobinshe@gmail.com>
> wrote:
> >
> >> Todd,
> >>
> >> Actually I'm wondering how kafka handle so much partition, with one
> >> partition there is at least one file on disk, and with 4000 partition,
> >> there will be at least 4000 files.
> >>
> >> When all these partitions have write request, how did Kafka make the
> write
> >> operation on the disk to be sequential (which is emphasized in the
> design
> >> document of Kafka) and make sure the disk access is effective?
> >>
> >> Thank you for your reply.
> >>
> >> xiaobinshe
> >>
> >>
> >>
> >> 2014-10-22 5:10 GMT+08:00 Todd Palino <tpalino@gmail.com>:
> >>
> >> > As far as the number of partitions a single broker can handle, we've
> set
> >> > our cap at 4000 partitions (including replicas). Above that we've seen
> >> some
> >> > performance and stability issues.
> >> >
> >> > -Todd
> >> >
> >> > On Tue, Oct 21, 2014 at 12:15 AM, Xiaobin She <xiaobinshe@gmail.com>
> >> > wrote:
> >> >
> >> > > hello, everyone
> >> > >
> >> > > I'm new to kafka, I'm wondering what's the max num of partition can
> one
> >> > > siggle machine handle in Kafka?
> >> > >
> >> > > Is there an sugeest num?
> >> > >
> >> > > Thanks.
> >> > >
> >> > > xiaobinshe
> >> > >
> >> >
> >>
>



-- 
the sun shines for all

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message