kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boyang Chen <reluctanthero...@gmail.com>
Subject Re: How do I tell Kafka Streams not to repartition?
Date Fri, 09 Aug 2019 22:27:21 GMT
Hey Tim,

I think the functionality you need is groupByKey() which avoids
repartitioning, feel free to check it out here:
https://docs.confluent.io/current/streams/developer-guide/dsl-api.html#aggregating.
Recommend you to read the whole thing but feel free just to search
`groupByKey`.

On Fri, Aug 9, 2019 at 7:14 AM Tim Ward <tim.ward@origamienergy.com> wrote:

> I've got an input topic which is keyed by "parent ID". Each message
> contains multiple items of data, each for a different "child ID".
>
> To process these items separately I flatMapValues() the stream to make a
> new stream of the inner items of data, keyed by "child ID".
>
> Now, because I've changed the key, Kafka Streams thinks a repartition is
> needed. But in fact it isn't, because all the inner items for a particular
> "child ID" will be contained within messages keyed with the same "parent
> ID".
>
> How do I tell Kafka Streams that there is no need to repartition in this
> case, because all the data that should remain together in the same instance
> of the application will do so without repartitioning? (I appreciate that
> Streams can't know about the parent-child relationship unless I *do* tell
> it in some way.)
>
> Tim Ward
>
> This email is from Origami Energy Limited. The contents of this email and
> any attachment are confidential to the intended recipient(s). If you are
> not an intended recipient: (i) do not use, disclose, distribute, copy or
> publish this email or its contents; (ii) please contact Origami Energy
> Limited immediately; and then (iii) delete this email. For more
> information, our privacy policy is available here:
> https://origamienergy.com/privacy-policy/. Origami Energy Limited
> (company number 8619644) is a company registered in England with its
> registered office at Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message