kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Liu <danyong...@gmail.com>
Subject kafka 0.8.1 socket leak? (file descriptor outage)
Date Fri, 28 Nov 2014 03:17:39 GMT
Hi,

We are using kafka 0.8.1 and it's working well for a long time. But there
is an issue comes up two days ago:

We monitored that kafka is not accepting new messages and get following
exception. It's complaining that too many open files.

We did a count on the socket used by the kafka process via "ls -ltr
/proc/6770/fd | grep "socket" | wc -l" and noticed that the fds reached the
limit (4096).

Then we tried to recover the kafka by restart the kafka-server, we continue
monitoring the sockets used by kafka and found that there will be 7 new
sockets created every 30 seconds.

Below trace confirmed that the socket increase happens every 30 seconds:


[2014-11-27 21:48:05,268] TRACE Finished writing, registering for read on
connection /10.158.141.31:45648 (kafka.network.Processor)
[2014-11-27 21:48:05,269] TRACE Finished writing, registering for read on
connection /10.158.141.31:45647 (kafka.network.Processor)
[2014-11-27 21:48:05,271] TRACE Finished writing, registering for read on
connection /10.158.141.31:45650 (kafka.network.Processor)
[2014-11-27 21:48:05,272] TRACE Finished writing, registering for read on
connection /10.158.141.31:45649 (kafka.network.Processor)
[2014-11-27 21:48:05,279] TRACE Finished writing, registering for read on
connection /10.158.141.31:45652 (kafka.network.Processor)
[2014-11-27 21:48:05,281] TRACE Finished writing, registering for read on
connection /10.158.141.31:45651 (kafka.network.Processor)
[2014-11-27 21:48:05,281] TRACE Finished writing, registering for read on
connection /10.158.141.31:45653 (kafka.network.Processor)
[2014-11-27 21:48:35,305] TRACE Finished writing, registering for read on
connection /10.158.141.31:45654 (kafka.network.Processor)
[2014-11-27 21:48:35,305] TRACE Finished writing, registering for read on
connection /10.158.141.31:45656 (kafka.network.Processor)
[2014-11-27 21:48:35,307] TRACE Finished writing, registering for read on
connection /10.158.141.31:45655 (kafka.network.Processor)
[2014-11-27 21:48:35,310] TRACE Finished writing, registering for read on
connection /10.158.141.31:45657 (kafka.network.Processor)
[2014-11-27 21:48:35,311] TRACE Finished writing, registering for read on
connection /10.158.141.31:45659 (kafka.network.Processor)
[2014-11-27 21:48:35,314] TRACE Finished writing, registering for read on
connection /10.158.141.31:45660 (kafka.network.Processor)
[2014-11-27 21:48:35,314] TRACE Finished writing, registering for read on
connection /10.158.141.31:45658 (kafka.network.Processor)
[2014-11-27 21:49:05,309] TRACE Finished writing, registering for read on
connection /10.158.141.31:45662 (kafka.network.Processor)
[2014-11-27 21:49:05,310] TRACE Finished writing, registering for read on
connection /10.158.141.31:45661 (kafka.network.Processor)
[2014-11-27 21:49:05,321] TRACE Finished writing, registering for read on
connection /10.158.141.31:45663 (kafka.network.Processor)
[2014-11-27 21:49:05,323] TRACE Finished writing, registering for read on
connection /10.158.141.31:45664 (kafka.network.Processor)
[2014-11-27 21:49:05,324] TRACE Finished writing, registering for read on
connection /10.158.141.31:45665 (kafka.network.Processor)
[2014-11-27 21:49:05,326] TRACE Finished writing, registering for read on
connection /10.158.141.31:45666 (kafka.network.Processor)
[2014-11-27 21:49:05,326] TRACE Finished writing, registering for read on
connection /10.158.141.31:45667 (kafka.network.Processor)

Does this a known issue?

Attached the full log.

-- 
Regards,
Daniel Liu

Mime
View raw message