kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Ward <tim.w...@origamienergy.com>
Subject How do I tell Kafka Streams not to repartition?
Date Fri, 09 Aug 2019 14:13:52 GMT
I've got an input topic which is keyed by "parent ID". Each message contains multiple items
of data, each for a different "child ID".

To process these items separately I flatMapValues() the stream to make a new stream of the
inner items of data, keyed by "child ID".

Now, because I've changed the key, Kafka Streams thinks a repartition is needed. But in fact
it isn't, because all the inner items for a particular "child ID" will be contained within
messages keyed with the same "parent ID".

How do I tell Kafka Streams that there is no need to repartition in this case, because all
the data that should remain together in the same instance of the application will do so without
repartitioning? (I appreciate that Streams can't know about the parent-child relationship
unless I *do* tell it in some way.)

Tim Ward

This email is from Origami Energy Limited. The contents of this email and any attachment are
confidential to the intended recipient(s). If you are not an intended recipient: (i) do not
use, disclose, distribute, copy or publish this email or its contents; (ii) please contact
Origami Energy Limited immediately; and then (iii) delete this email. For more information,
our privacy policy is available here: https://origamienergy.com/privacy-policy/. Origami Energy
Limited (company number 8619644) is a company registered in England with its registered office
at Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message