kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: kafka 0.8.1 socket leak? (file descriptor outage)
Date Sat, 29 Nov 2014 16:41:14 GMT
These logs don't indicate new socket connections. You need to look for
debug logs like "Accepted connection from". Are those connections from a
producer or a consumer? Is it possible that a client didn't call close()
properly?

Thanks,

Jun

On Thu, Nov 27, 2014 at 7:17 PM, Daniel Liu <danyongliu@gmail.com> wrote:

> Hi,
>
> We are using kafka 0.8.1 and it's working well for a long time. But there
> is an issue comes up two days ago:
>
> We monitored that kafka is not accepting new messages and get following
> exception. It's complaining that too many open files.
>
> We did a count on the socket used by the kafka process via "ls -ltr
> /proc/6770/fd | grep "socket" | wc -l" and noticed that the fds reached the
> limit (4096).
>
> Then we tried to recover the kafka by restart the kafka-server, we
> continue monitoring the sockets used by kafka and found that there will be
> 7 new sockets created every 30 seconds.
>
> Below trace confirmed that the socket increase happens every 30 seconds:
>
>
> [2014-11-27 21:48:05,268] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45648 (kafka.network.Processor)
> [2014-11-27 21:48:05,269] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45647 (kafka.network.Processor)
> [2014-11-27 21:48:05,271] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45650 (kafka.network.Processor)
> [2014-11-27 21:48:05,272] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45649 (kafka.network.Processor)
> [2014-11-27 21:48:05,279] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45652 (kafka.network.Processor)
> [2014-11-27 21:48:05,281] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45651 (kafka.network.Processor)
> [2014-11-27 21:48:05,281] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45653 (kafka.network.Processor)
> [2014-11-27 21:48:35,305] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45654 (kafka.network.Processor)
> [2014-11-27 21:48:35,305] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45656 (kafka.network.Processor)
> [2014-11-27 21:48:35,307] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45655 (kafka.network.Processor)
> [2014-11-27 21:48:35,310] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45657 (kafka.network.Processor)
> [2014-11-27 21:48:35,311] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45659 (kafka.network.Processor)
> [2014-11-27 21:48:35,314] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45660 (kafka.network.Processor)
> [2014-11-27 21:48:35,314] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45658 (kafka.network.Processor)
> [2014-11-27 21:49:05,309] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45662 (kafka.network.Processor)
> [2014-11-27 21:49:05,310] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45661 (kafka.network.Processor)
> [2014-11-27 21:49:05,321] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45663 (kafka.network.Processor)
> [2014-11-27 21:49:05,323] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45664 (kafka.network.Processor)
> [2014-11-27 21:49:05,324] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45665 (kafka.network.Processor)
> [2014-11-27 21:49:05,326] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45666 (kafka.network.Processor)
> [2014-11-27 21:49:05,326] TRACE Finished writing, registering for read on
> connection /10.158.141.31:45667 (kafka.network.Processor)
>
> Does this a known issue?
>
> Attached the full log.
>
> --
> Regards,
> Daniel Liu
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message