kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: log.retention.size
Date Fri, 02 May 2014 14:49:34 GMT
log.retention.size controls the total size in a log dir (per
partition). log.file.size
controls the size of each log segment in the log dir.

Thanks,

Jun


On Thu, May 1, 2014 at 9:31 PM, vinh <vinh@loggly.com> wrote:

> In the 0.7 docs, the description for log.retention.size and log.file.size
> sound very much the same.  In particular, that they apply to a single log
> file (or log segment file).
>
> http://kafka.apache.org/07/configuration.html
>
> I'm beginning to think there is no setting to control the max aggregate
> size of all logs.  If this is correct, what would be a good approach to
> enforce this requirement?  In my particular scenario, I have a lot of data
> being written to Kafka at a very high rate.  So a 1TB disk can easily be
> filled up in 24hrs or so.  One option is to add more Kafka brokers to add
> more disk space to the pool, but I'd like to avoid that and see if I can
> simply configure Kafka to not write more than 1TB aggregate.  Else, Kafka
> will OOM and kill itself, and possibly the crash the node itself because
> the disk is full.
>
>
> On May 1, 2014, at 9:21 PM, vinh <vinh@loggly.com> wrote:
>
> > Using Kafka 0.7.2, I have the following in server.properties:
> >
> > log.retention.hours=48
> > log.retention.size=107374182400
> > log.file.size=536870912
> >
> > My interpretation of this is:
> > a) a single log segment file over 48hrs old will be deleted
> > b) the total combined size of *all* logs is 100GB
> > c) a single log segment file is limited to 500MB in size before a new
> segment file is spawned spawning a new segment file
> > d) a "log file" can be composed of many "log segment files"
> >
> > But, even after setting the above, I find that the total combined size
> of all Kafka logs on disk is 200GB right now.  Isn't log.retention.size
> supposed to limit it to 100GB?  Am I missing something?  The docs are not
> really clear, especially when it comes to distinguishing between a "log
> file" and a "log segment file".
> >
> > I have disk monitoring.  But like anything else in software, even
> monitoring can fail.  Via configuration, I'd like to make sure that Kafka
> does not write more than the available disk space.  Or something like
> log4j, where I can set a max number of log files and the max size per file,
> which essentially allows me to set a max aggregate size limit across all
> logs.
> >
> > Thanks,
> > -Vinh
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message