kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Björn Häuser <bjoernhaeu...@gmail.com>
Subject Re: Kafka Streams Internal Topic Retention not applied
Date Fri, 06 Apr 2018 21:10:26 GMT
Hello Guozhang

thanks.

So after reading much more docs I still do not have the complete picture.

These are our relevant settings from kafka broker configuration:

log.cleanup.policy=delete
# set log.retention.bytes to 15 gb
log.retention.bytes=16106127360
# set log.retention.hours to 30 days
log.retention.hours=720

Though one of internal kafka stream topics (with cleanup.policy = compact) grew to ~40gb today.


What am I missing? I thought these settings should take care that a given topic is never >15
gb, right?

Thanks
Björn

> On 29. Mar 2018, at 00:05, Guozhang Wang <wangguoz@gmail.com> wrote:
> 
> Hello,
> 
> You can set the topic-level configs via the
> StreamsConfig#topicPrefix(String), please find the following web docs
> (search for KIP-173):
> 
> https://kafka.apache.org/documentation/streams/upgrade-guide#streams_api_changes_100
> 
> 
> Guozhang
> 
> 
> 
> On Wed, Mar 28, 2018 at 3:23 AM, Björn Häuser <bjoernhaeuser@gmail.com>
> wrote:
> 
>> Hello Everyone,
>> 
>> we are running a Kafka Streams Application with does time window
>> aggregates (using kafka 1.0.0).
>> 
>> Unfortunately one of the changelog topics is now growing quite a bit in
>> size maxing out the brokers. I did not find any settings in the kafka
>> stream properties to configure retention and went ahead and set it
>> retention.bytes to 15gb. Unfortunately this does not seem to apply and the
>> topic size is still around 140gb.
>> 
>> Is this intended? I could not find any documentation about setting the
>> retention size for the internal topics.
>> 
>> Thanks
>> Björn
>> 
>> 
>> 
> 
> 
> -- 
> -- Guozhang


Mime
View raw message