kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vinh <v...@loggly.com>
Subject Re: log.retention.size
Date Fri, 02 May 2014 04:31:10 GMT
In the 0.7 docs, the description for log.retention.size and log.file.size sound very much the
same.  In particular, that they apply to a single log file (or log segment file).

http://kafka.apache.org/07/configuration.html

I'm beginning to think there is no setting to control the max aggregate size of all logs.
 If this is correct, what would be a good approach to enforce this requirement?  In my particular
scenario, I have a lot of data being written to Kafka at a very high rate.  So a 1TB disk
can easily be filled up in 24hrs or so.  One option is to add more Kafka brokers to add more
disk space to the pool, but I'd like to avoid that and see if I can simply configure Kafka
to not write more than 1TB aggregate.  Else, Kafka will OOM and kill itself, and possibly
the crash the node itself because the disk is full.


On May 1, 2014, at 9:21 PM, vinh <vinh@loggly.com> wrote:

> Using Kafka 0.7.2, I have the following in server.properties:
> 
> log.retention.hours=48
> log.retention.size=107374182400
> log.file.size=536870912
> 
> My interpretation of this is:
> a) a single log segment file over 48hrs old will be deleted
> b) the total combined size of *all* logs is 100GB
> c) a single log segment file is limited to 500MB in size before a new segment file is
spawned spawning a new segment file
> d) a "log file" can be composed of many "log segment files"
> 
> But, even after setting the above, I find that the total combined size of all Kafka logs
on disk is 200GB right now.  Isn't log.retention.size supposed to limit it to 100GB?  Am I
missing something?  The docs are not really clear, especially when it comes to distinguishing
between a "log file" and a "log segment file".
> 
> I have disk monitoring.  But like anything else in software, even monitoring can fail.
 Via configuration, I'd like to make sure that Kafka does not write more than the available
disk space.  Or something like log4j, where I can set a max number of log files and the max
size per file, which essentially allows me to set a max aggregate size limit across all logs.
> 
> Thanks,
> -Vinh


Mime
View raw message