kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyler Monahan <tjmonah...@gmail.com>
Subject Re: 90 open file handles for one segment
Date Tue, 11 Sep 2018 20:04:04 GMT
Matt,


I am seeing similar behavior with kafka 1.1.0 with 80 copies of the same
file being open. I am getting up to ~500,000 open files on each broker with
it dying around 600,000. I have yet to find any documentation saying how
many open files kafka should have per topic/partition. The only
documentation I have seen is having the limit set to at least 100,000 which
we are both well past.


[2018-09-07 14:35:33,872] ERROR Error while accepting connection
(kafka.network.Acceptor)

java.io.IOException: Too many open files


Tyler Monahan


´╗┐On 8/27/18, 12:29 PM, "Matt Kocubinski" <mkocubinski@gmail.com> wrote:



    A 5 node cluster is running Kafka 1.0.0 with 126 topics, most of them
with

    24 partitions.  Brokers are keeping around 800k file handles open each,
and

    regularly crashing with "Too many open file handles".



    Using one broker as a study I discovered There are ~13k log file

    segments/indices in the Kafka data dir discovered like:



    admin@kafka-1:/kafka/logs$ find . -type f | wc -l

    12368



     I've also noticed that a single segment file is open in 90 different

    threads:



    admin@kafka-1:/kafka/logs$ sudo lsof | grep

    "/site.time.tracking-6/00000000000000000297.log" | wc -l

    90



    This is about 1 per broker thread running!  Extrapolating this nearly
adds

    up to 1.3mm, or the 800k I'm seeing.  Something must be grossly

    mis-configured here.  Why are the same segments being opened so many
times?

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message