kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Gordon <jonathangor...@newrelic.com.INVALID>
Subject Kafka Streams Session store performance degradation from 0.10.2.1 to 0.11.0.3
Date Tue, 06 Nov 2018 17:49:19 GMT
I have a Kafka Streams app that I'm trying to upgrade from 0.10.2.1 to
0.11.0.3 but when I do I notice that CPU goes way up and consumption goes
down. A thread profile indicates that the most expensive task is during our
aggregation, fetching from the cache.

Thread profile with caching:
https://imgur.com/l5VEsC2

If I disable the cache both performance and consumption are good but we are
producing every single aggregation modification, which is not what we want.

Thread profile without caching:
https://imgur.com/a/JK3nkou

I read this thread, which seems relevant e

https://lists.apache.org/thread.html/2b44e74eaec7172b107bcff96861cf8b4837f55a44714f69d033cc2e@%3Cusers.kafka.apache.org%3E

Notably: "Note, that caching was _not_ introduced to reduce the writes to
RocksDB, but to reduce the write the the changelog topic and to reduce the
number of records send downstream."

So how can we reduce the number of records sent downstream while
maintaining the same performance characteristics that we have with caching
turned off? Or put another way, how can I upgrade my app without taking a
hit in performance or behavior?

Thanks!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message