kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anirudh P <panirudh2...@gmail.com>
Subject Re: Too Many Open Files
Date Mon, 01 Aug 2016 08:04:18 GMT
I agree with Steve. We had a similar problem where we set the ulimit to a
certain value but it was getting overridden.
It only worked when we set the ulimit after logging in as root. You might
want to give that a try if you have not done so already

- Anirudh

On Mon, Aug 1, 2016 at 1:19 PM, Steve Miller <steve@idrathernotsay.com>
wrote:

> Can you run lsof -p (pid) for whatever the pid is for your Kafka process?
>
> For the fd limits you've set, I don't think subtlety is required: if
> there's a millionish lines in the output, the fd limit you set is where you
> think it is, and if it's a lot lower than that, the limit isn't being
> applied properly somehow (maybe you are running this under, say,
> supervisord, and maybe its config is lowering the limit, or the limits for
> root are as you say but the limits for the kafka user aren't being set
> properly, that sort of thing).
>
> If you do have 1M lines in the output, at least this might give you a
> place to start figuring out what's open and why.
>
>     -Steve
>
> > On Jul 31, 2016, at 4:14 PM, Kessiler Rodrigues <kessiler@callinize.com>
> wrote:
> >
> > I’m still experiencing this issue…
> >
> > Here are the kafka logs.
> >
> > [2016-07-31 20:10:35,658] ERROR Error while accepting connection
> (kafka.network.Acceptor)
> > java.io.IOException: Too many open files
> >    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> >    at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> >    at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> >    at kafka.network.Acceptor.accept(SocketServer.scala:323)
> >    at kafka.network.Acceptor.run(SocketServer.scala:268)
> >    at java.lang.Thread.run(Thread.java:745)
> > [2016-07-31 20:10:35,658] ERROR Error while accepting connection
> (kafka.network.Acceptor)
> > java.io.IOException: Too many open files
> >    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> >    at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> >    at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> >    at kafka.network.Acceptor.accept(SocketServer.scala:323)
> >    at kafka.network.Acceptor.run(SocketServer.scala:268)
> >    at java.lang.Thread.run(Thread.java:745)
> > [2016-07-31 20:10:35,658] ERROR Error while accepting connection
> (kafka.network.Acceptor)
> > java.io.IOException: Too many open files
> >    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> >    at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> >    at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> >    at kafka.network.Acceptor.accept(SocketServer.scala:323)
> >    at kafka.network.Acceptor.run(SocketServer.scala:268)
> >    at java.lang.Thread.run(Thread.java:745)
> >
> > My ulimit is 1 million, how is that possible?
> >
> > Can someone help with this?
> >
> >
> >> On Jul 30, 2016, at 5:05 AM, Kessiler Rodrigues <kessiler@callinize.com>
> wrote:
> >>
> >> I have changed it a bit.
> >>
> >> I have 10 brokers and 20k topics with 1 partition each.
> >>
> >> I looked at the kaka’s logs dir and I only have 3318 files.
> >>
> >> I’m doing some tests to see how many topics/partitions I can have, but
> it is throwing too many files once it hits 15k topics..
> >>
> >> Any thoughts?
> >>
> >>
> >>
> >>> On Jul 29, 2016, at 10:33 PM, Gwen Shapira <gwen@confluent.io> wrote:
> >>>
> >>> woah, it looks like you have 15,000 replicas per broker?
> >>>
> >>> You can go into the directory you configured for kafka's log.dir and
> >>> see how many files you have there. Depending on your segment size and
> >>> retention policy, you could have hundreds of files per partition
> >>> there...
> >>>
> >>> Make sure you have at least that many file handles and then also add
> >>> handles for the client connections.
> >>>
> >>> 1 million file handles sound like a lot, but you are running lots of
> >>> partitions per broker...
> >>>
> >>> We normally don't see more than maybe 4000 per broker and most
> >>> clusters have a lot fewer, so consider adding brokers and spreading
> >>> partitions around a bit.
> >>>
> >>> Gwen
> >>>
> >>> On Fri, Jul 29, 2016 at 12:00 PM, Kessiler Rodrigues
> >>> <kessiler@callinize.com> wrote:
> >>>> Hi guys,
> >>>>
> >>>> I have been experiencing some issues on kafka, where its throwing too
> many open files.
> >>>>
> >>>> I have around of 6k topics and 5 partitions each.
> >>>>
> >>>> My cluster was made with 6 brokers. All of them are running Ubuntu 16
> and the file limits settings are:
> >>>>
> >>>> `cat  /proc/sys/fs/file-max`
> >>>> 2000000
> >>>>
> >>>> `ulimit -n`
> >>>> 1000000
> >>>>
> >>>> Anyone has experienced it before?
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message