kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kaufman Ng <kauf...@confluent.io>
Subject Re: why more open files
Date Wed, 05 Jul 2017 14:30:25 GMT
Keep in mind Kafka brokers can use many file descriptors/handles. You may
need to increase the OS file descriptor limits.

http://kafka.apache.org/documentation/#os

"File descriptor limits: Kafka uses file descriptors for log segments and
open connections. If a broker hosts many partitions, consider that the
broker needs at least (number_of_partitions)*(partition_size/segment_size)
to track all log segments in addition to the number of connections the
broker makes. We recommend at least 100000 allowed file descriptors for the
broker processes as a starting point."

On Wed, Jul 5, 2017 at 10:24 AM, M. Manna <manmedia@gmail.com> wrote:

> This is quite vague.
>
> What commands have you executed?
>
> What do you refer to by open files? Is it the log partition or consumer
> offsets?
>
> On 5 Jul 2017 3:21 pm, "Satyavathi Anasuri" <satyavathi.a@imimobile.com>
> wrote:
>
> > Hi,
> >                    I have created a topic with 500 partitions in 3 node
> > cluster with replication factor 3. kafka version is 0.11. I executed lsof
> > command and it lists more 1 lakh open files. why these many open files
> and
> > how to reduce it ?.
> > reg's
> > Satya.
> >
>



-- 
Kaufman Ng
+1 646 961 8063
Solutions Architect | Confluent | www.confluent.io

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message