kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruno Cadonna <br...@confluent.io>
Subject Re: Merging create/delete updates for Kafka Streams aggregations
Date Fri, 15 Nov 2019 11:36:53 GMT
Hi Thilo,

You can influence the rate of updates of aggregations by configuring
the size of the record caches with `cache.max.bytes.buffering`.

Details can be found here:
https://kafka.apache.org/22/documentation/streams/developer-guide/dsl-api.html#aggregating
https://kafka.apache.org/22/documentation/streams/developer-guide/memory-mgmt.html#streams-developer-guide-memory-management-record-cache

If you aggregate over windows, you should have a look at `suppress()`.

Details can be found here:
 https://kafka.apache.org/22/documentation/streams/developer-guide/dsl-api.html#window-final-results

Best,
Bruno

On Fri, Nov 15, 2019 at 12:02 PM Thilo-Alexander Ginkel
<thilo@ginkel.com> wrote:
>
> Hello everyone,
>
> we are using the Kafka Streams DSL (v2.2.1) to perform aggregations on a
> topic using org.apache.kafka.streams.kstream.KGroupedTable#aggregate. ATM
> we are seeing one update being published when the subtractor is being
> called and another one when the adder is called.
>
> I was under the impression that there is some setting that allows
> folding/merging these updates for most cases.
>
> As I can't seem to find this in the docs, can someone please help me out
> with a pointer?
>
> Thanks & kind regards,
> Thilo

Mime
View raw message