kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Krahmer <bkrah...@krahmer.com>
Subject Re: Streams - merging multiple topics
Date Tue, 22 Nov 2016 15:44:08 GMT
Thanks Damian!  Based on your response, I finally got it working.  I did 
end up using left joins and added a final step that goes from table -> 
stream and then filters out nulls.

thanks,
brian


On 21.11.2016 22:03, Damian Guy wrote:
> Hi Brian,
>
> It sounds like you might want do something like:
>
> KTable inputOne = builder.table("input-one");
> KTable inputTwo = builder.table("input-two");
> KTable inputThree = builder.table("input-three");
> ValueJoiner joiner1 = //...
> ValueJoiner joiner2 = //...
>
> inputOne.join(inputTwo, joiner1)
>                 .join(inputThree, joiner2)
>                 .to(outputTopic)
>
> This would result in the join logic being triggered when any record arrives
> in any of the input topics. There will be some de-duplication of results
> written to the output topic. If you want immediate output then you will
> want to set StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG = 0. You should
> create and configure the output topic yourself before you run your streams
> app (so you can make it compacted etc).
>
> Thanks,
> Damian
>
>


Mime
View raw message