kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vincent Maurin <vincent.mau...@glispa.com>
Subject Is the key for a state need to be the key of incoming message
Date Wed, 24 Jan 2018 08:07:54 GMT
Hello,

I am building a kafka stream application consuming a log compacted topic of
12 partitions. Each message has a String key and a json body, the json body
contains a date field.
I have made a custom Transformer in my topology that consumes this stream,
immediately forward the document where the date is in the past and keep in
the state the documents where the date is in the future. Then with a
Ponctuator, I am looking to the state and eventually forwarding documents
when the date is changing.
The current key I use for my state is the date, and it is not related to
the message key. Is it problematic ? I would like to avoid repartitioning
by date because it would be unbalanced

Best,
Vincent

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message