kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Richardson <ch...@chrisrichardson.net>
Subject Re: Too Many Open Files
Date Sun, 31 Jul 2016 21:26:38 GMT
Gwen,

Is there any particular reason why "inactive" (no consumers or producers
for a topic)  files need to be open?

Chris

-- 
Learn microservices - http://learnmicroservices.io
Microservices application platform http://eventuate.io

On Fri, Jul 29, 2016 at 6:33 PM, Gwen Shapira <gwen@confluent.io> wrote:

> woah, it looks like you have 15,000 replicas per broker?
>
> You can go into the directory you configured for kafka's log.dir and
> see how many files you have there. Depending on your segment size and
> retention policy, you could have hundreds of files per partition
> there...
>
> Make sure you have at least that many file handles and then also add
> handles for the client connections.
>
> 1 million file handles sound like a lot, but you are running lots of
> partitions per broker...
>
> We normally don't see more than maybe 4000 per broker and most
> clusters have a lot fewer, so consider adding brokers and spreading
> partitions around a bit.
>
> Gwen
>
> On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues
> <kessiler@callinize.com> wrote:
> > Hi guys,
> >
> > I have been experiencing some issues on kafka, where its throwing too
> many open files.
> >
> > I have around of 6k topics and 5 partitions each.
> >
> > My cluster was made with 6 brokers. All of them are running Ubuntu 16
> and the file limits settings are:
> >
> > `cat  /proc/sys/fs/file-max`
> > 2000000
> >
> >  `ulimit -n`
> > 1000000
> >
> > Anyone has experienced it before?
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message