kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daisuke Moriya <dmor...@yahoo-corp.jp>
Subject changelog topic with in-memory state store
Date Wed, 11 Jan 2017 06:10:07 GMT
Hi.

I am developing a simple log counting application using Kafka Streams 0.10.1.1.
Its implementation is almost the same as the WordCountProcessor in the confluent document
[http://docs.confluent.io/3.1.1/streams/developer-guide.html#processor-api].
I am using in-memory state store, 
its key is the ID of log category, value is count.
All the changelogs are written to a broker by context.commit() for fault tolerance,
but since the data I handle is large and the size of key is large, it takes a long time to
process.
Even if it is compacted with a broker, this will put a load on the broker.
I would like to write only the latest records for each key on the broker 
instead of all changelogs at context.commit(). 
This will reduce the load on the broker and I do not think 
there will be any negative impact on fault tolerance.
If I use the persistent state store, I can do this by enabling caching, 
but I couldn't find how to accomplish this with the in-memory state store.
Can I do this?

Thank you,
--
Daisuke



Mime
View raw message