kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Kafka Streams Session store performance degradation from 0.10.2.1 to 0.11.0.3
Date Tue, 06 Nov 2018 19:22:16 GMT
Not sure atm why you see a performance degradation. Would need to dig
into the details.

However, did you consider to upgrade to 2.0 instead or 0.11?

Also note that we added a new operator `suppress()` in upcoming 2.1
release, that allows you to do rate control without caching:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables

Hope this helps.


-Matthias

On 11/6/18 9:49 AM, Jonathan Gordon wrote:
> I have a Kafka Streams app that I'm trying to upgrade from 0.10.2.1 to
> 0.11.0.3 but when I do I notice that CPU goes way up and consumption goes
> down. A thread profile indicates that the most expensive task is during our
> aggregation, fetching from the cache.
> 
> Thread profile with caching:
> https://imgur.com/l5VEsC2
> 
> If I disable the cache both performance and consumption are good but we are
> producing every single aggregation modification, which is not what we want.
> 
> Thread profile without caching:
> https://imgur.com/a/JK3nkou
> 
> I read this thread, which seems relevant e
> 
> https://lists.apache.org/thread.html/2b44e74eaec7172b107bcff96861cf8b4837f55a44714f69d033cc2e@%3Cusers.kafka.apache.org%3E
> 
> Notably: "Note, that caching was _not_ introduced to reduce the writes to
> RocksDB, but to reduce the write the the changelog topic and to reduce the
> number of records send downstream."
> 
> So how can we reduce the number of records sent downstream while
> maintaining the same performance characteristics that we have with caching
> turned off? Or put another way, how can I upgrade my app without taking a
> hit in performance or behavior?
> 
> Thanks!
> 


Mime
View raw message