kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lung, Paul" <pl...@ebay.com>
Subject Re: Getting "java.io.IOException: Too many open files"
Date Wed, 25 Jun 2014 05:51:06 GMT
The controller machine has 3500 or so, while the other machines have
around 1600.

Paul Lung

On 6/24/14, 10:31 PM, "Prakash Gowri Shankor" <prakash.shankor@gmail.com>
wrote:

>How many files does each broker itself have open ? You can find this from
>'ls -l /proc/<processid>/fd'
>
>
>
>
>On Tue, Jun 24, 2014 at 10:18 PM, Lung, Paul <plung@ebay.com> wrote:
>
>> Hi All,
>>
>>
>> I just upgraded my cluster from 0.8.1 to 0.8.1.1. I¹m seeing the
>>following
>> error messages on the same 3 brokers once in a while:
>>
>>
>> [2014-06-24 21:43:44,711] ERROR Error in acceptor
>>(kafka.network.Acceptor)
>>
>> java.io.IOException: Too many open files
>>
>>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>
>>         at
>> 
>>sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:16
>>3)
>>
>>         at kafka.network.Acceptor.accept(SocketServer.scala:200)
>>
>>         at kafka.network.Acceptor.run(SocketServer.scala:154)
>>
>>         at java.lang.Thread.run(Thread.java:679)
>>
>> [2014-06-24 21:43:44,711] ERROR Error in acceptor
>>(kafka.network.Acceptor)
>>
>> java.io.IOException: Too many open files
>>
>>         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>
>>         at
>> 
>>sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:16
>>3)
>>
>>         at kafka.network.Acceptor.accept(SocketServer.scala:200)
>>
>>         at kafka.network.Acceptor.run(SocketServer.scala:154)
>>
>>         at java.lang.Thread.run(Thread.java:679)
>>
>> When this happens, these 3 brokers essentially go out of sync when you
>>do
>> a ³kafka-topics.sh ‹describe².
>>
>> I tracked the number of open files by doing ³watch ­n 1 Œsudo lsof | wc
>> ­l¹², which basically counts all open files on the system. The numbers
>>for
>> the systems are basically in the 6000 range, with one system going to
>>9000.
>> I presume the 9000 machine is the controller. Looking at the ulimit of
>>the
>> user, both the hard limit and the soft limit for open files is 100,000.
>> Using sysctl, the max file is fs.file-max = 9774928. So we seem to be
>>way
>> under the limit.
>>
>> What am I missing here? Is there some JVM limit around 10K open files or
>> something?
>>
>> Paul Lung
>>


Mime
View raw message