kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lung, Paul" <pl...@ebay.com>
Subject Re: Getting "java.io.IOException: Too many open files"
Date Wed, 25 Jun 2014 17:19:21 GMT
Hi Prakash,

How many open files do you expect a broker to be able to handle? It seems
like this broker is crashing at around 4100 or so open files.

Thanks,
Paul Lung

On 6/24/14, 11:08 PM, "Lung, Paul" <plung@ebay.com> wrote:

>Ok. What I just saw was that when the controller machine reaches around
>4100+ files, it crashes. Then I think the controller bounced between 2
>other machines, taking them down too, and the circled back to the original
>machine.
>
>Paul Lung
>
>On 6/24/14, 10:51 PM, "Lung, Paul" <plung@ebay.com> wrote:
>
>>The controller machine has 3500 or so, while the other machines have
>>around 1600.
>>
>>Paul Lung
>>
>>On 6/24/14, 10:31 PM, "Prakash Gowri Shankor" <prakash.shankor@gmail.com>
>>wrote:
>>
>>>How many files does each broker itself have open ? You can find this
>>>from
>>>'ls -l /proc/<processid>/fd'
>>>
>>>
>>>
>>>
>>>On Tue, Jun 24, 2014 at 10:18 PM, Lung, Paul <plung@ebay.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>>
>>>> I just upgraded my cluster from 0.8.1 to 0.8.1.1. I¹m seeing the
>>>>following
>>>> error messages on the same 3 brokers once in a while:
>>>>
>>>>
>>>> [2014-06-24 21:43:44,711] ERROR Error in acceptor
>>>>(kafka.network.Acceptor)
>>>>
>>>> java.io.IOException: Too many open files
>>>>
>>>>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>>
>>>>         at
>>>> 
>>>>sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:
>>>>1
>>>>6
>>>>3)
>>>>
>>>>         at kafka.network.Acceptor.accept(SocketServer.scala:200)
>>>>
>>>>         at kafka.network.Acceptor.run(SocketServer.scala:154)
>>>>
>>>>         at java.lang.Thread.run(Thread.java:679)
>>>>
>>>> [2014-06-24 21:43:44,711] ERROR Error in acceptor
>>>>(kafka.network.Acceptor)
>>>>
>>>> java.io.IOException: Too many open files
>>>>
>>>>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>>
>>>>         at
>>>> 
>>>>sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:
>>>>1
>>>>6
>>>>3)
>>>>
>>>>         at kafka.network.Acceptor.accept(SocketServer.scala:200)
>>>>
>>>>         at kafka.network.Acceptor.run(SocketServer.scala:154)
>>>>
>>>>         at java.lang.Thread.run(Thread.java:679)
>>>>
>>>> When this happens, these 3 brokers essentially go out of sync when you
>>>>do
>>>> a ³kafka-topics.sh ‹describe².
>>>>
>>>> I tracked the number of open files by doing ³watch ­n 1 Œsudo lsof |
>>>>wc
>>>> ­l¹², which basically counts all open files on the system. The numbers
>>>>for
>>>> the systems are basically in the 6000 range, with one system going to
>>>>9000.
>>>> I presume the 9000 machine is the controller. Looking at the ulimit of
>>>>the
>>>> user, both the hard limit and the soft limit for open files is
>>>>100,000.
>>>> Using sysctl, the max file is fs.file-max = 9774928. So we seem to be
>>>>way
>>>> under the limit.
>>>>
>>>> What am I missing here? Is there some JVM limit around 10K open files
>>>>or
>>>> something?
>>>>
>>>> Paul Lung
>>>>
>>
>

Mime
View raw message