kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Klukas <jklu...@simple.com>
Subject How to "buffer" a stream with high churn and output only at the end of a window?
Date Tue, 19 Apr 2016 20:40:06 GMT
Is it true that the aggregation and reduction methods of KStream will emit
a new output message for each incoming message?

I have an application that's copying a Postgres replication stream to a
Kafka topic, and activity tends to be clustered, with many updates to a
given primary key happening in quick succession. I'd like to smooth that
out by buffering the messages in tumbling windows, allowing the updates to
overwrite one another, and emitting output messages only at the end of the
window.

Does the Kafka Streams API provide any hooks that I could use to achieve
this kind of windowed "buffering" or "deduplication" of a stream?

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message