kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Thibault <scott.thiba...@multiscalehn.com>
Subject Re: Broker holding all files open
Date Wed, 03 Jun 2015 21:00:57 GMT
This is great info.  Thanks Lance.

--Scott


On Wed, Jun 3, 2015 at 1:36 PM, Lance Laursen <llaursen@rubiconproject.com>
wrote:

> Hey Scott,
>
> There's not much you can do about this, other than increasing your
> log.segment.bytes (max 2GB) or lowering your partition counts on your
> mirrored cluster (probably not the best strategy unless you're dealing with
> 100,000+ small topics, at which point you should consider a single
> aggregate topic + key based partitioning instead). Kafka will keep a file
> handle open per log segment file and another per associated log segment
> index file. Your best bet is to permanently increase allowed open file
> descriptors, and run your log directory on XFS or other appropriate
> filesystem.
>
> Linux uses ~1KB of memory per open file descriptor. So, 100,000 handles =
> ~100MB. 100-200k open descriptors is not unusual for certain enterprisey
> apps, and 10-30k seems to be the norm if each of your brokers has a decent
> amount of disk space. A general recommendation exists to keep your file
> descriptor memory usage to 10% or lower than your total system memory, but
> this may be a bit arbitrary. The general thinking is that under normal use
> cases, an application should run into other resource constraints before any
> file descriptor constraint becomes an issue. If this is not true, you might
> consider re-evaluating your current design such that your partition count
> scales with your hardware resources & number of consumers rather than your
> external software design. As mentioned above, aggregate topics using
> key-based partitioning can help with this.
>
> Regards,
>
> On Wed, Jun 3, 2015 at 7:47 AM, Scott Thibault <
> scott.thibault@multiscalehn.com> wrote:
>
> > Hi,
> >
> > I'm running into the common issue of too many files open by the broker.
> > While increasing the open file limit is a short-term work around, I need
> a
> > long-term solution.  I have a Kafka mirror that is keeping log segments
> for
> > long periods of time and the number of files is potentially unbounded.
> >
> > Is there some way to prevent the broker from holding an open descriptor
> for
> > every file?
> >
> > --Scott Thibault
> >
> > --
> > *This e-mail is not encrypted.  Due to the unsecured nature of
> unencrypted
> > e-mail, there may be some level of risk that the information in this
> e-mail
> > could be read by a third party.  Accordingly, the recipient(s) named
> above
> > are hereby advised to not communicate protected health information using
> > this e-mail address.  If you desire to send protected health information
> > electronically, please contact MultiScale Health Networks at
> (206)538-6090
> > *
> >
>



-- 
*This e-mail is not encrypted.  Due to the unsecured nature of unencrypted
e-mail, there may be some level of risk that the information in this e-mail
could be read by a third party.  Accordingly, the recipient(s) named above
are hereby advised to not communicate protected health information using
this e-mail address.  If you desire to send protected health information
electronically, please contact MultiScale Health Networks at (206)538-6090*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message