kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Gordon <jonathangor...@newrelic.com.INVALID>
Subject Re: Kafka Streams Session store performance degradation from 0.10.2.1 to 0.11.0.3
Date Wed, 07 Nov 2018 23:28:21 GMT
Hi Matthias,

I upgraded to 2.0.0 and we're experiencing the same problem. I've posted a
new screengrab of a threadprofile:

https://imgur.com/a/2wncPHw

>From our perspective, it appears something happened after 0.10.2.1 that
made the LRU Cache much slower for our use case. What would you recommend
for our next steps?

Jonathan

On 2018/11/06 19:22:16, "Matthias J. Sax" <m...@confluent.io> wrote:
> Not sure atm why you see a performance degradation. Would need to dig>
> into the details.>
>
> However, did you consider to upgrade to 2.0 instead or 0.11?>
>
> Also note that we added a new operator `suppress()` in upcoming 2.1>
> release, that allows you to do rate control without caching:>
>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-328%3A+Ability+to+suppress+updates+for+KTables>

>
> Hope this helps.>
>
>
> -Matthias>
>
> On 11/6/18 9:49 AM, Jonathan Gordon wrote:>
> > I have a Kafka Streams app that I'm trying to upgrade from 0.10.2.1 to>
> > 0.11.0.3 but when I do I notice that CPU goes way up and consumption
goes>
> > down. A thread profile indicates that the most expensive task is during
our>
> > aggregation, fetching from the cache.>
> > >
> > Thread profile with caching:>
> > https://imgur.com/l5VEsC2>
> > >
> > If I disable the cache both performance and consumption are good but we
are>
> > producing every single aggregation modification, which is not what we
want.>
> > >
> > Thread profile without caching:>
> > https://imgur.com/a/JK3nkou>
> > >
> > I read this thread, which seems relevant e>
> > >
> >
https://lists.apache.org/thread.html/2b44e74eaec7172b107bcff96861cf8b4837f55a44714f69d033cc2e@%3Cusers.kafka.apache.org%3E>

> > >
> > Notably: "Note, that caching was _not_ introduced to reduce the writes
to>
> > RocksDB, but to reduce the write the the changelog topic and to reduce
the>
> > number of records send downstream.">
> > >
> > So how can we reduce the number of records sent downstream while>
> > maintaining the same performance characteristics that we have with
caching>
> > turned off? Or put another way, how can I upgrade my app without taking
a>
> > hit in performance or behavior?>
> > >
> > Thanks!>
> > >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message