kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Krahmer <bkrah...@krahmer.com>
Subject Streams - merging multiple topics
Date Mon, 21 Nov 2016 20:02:50 GMT
Hey guys. I've been banging my head for about 3 days now trying to get a 
streams application working with no luck. I've read through all of the 
docs and examples I can find, and just am not getting it.  I'm using 
0.10.1 and have worked quite a bit with the high-level consumer and 
publisher. What I would like to do is stream from 3 different topics 
(all keyed by the same uuid, with each having their own value type), and 
when a message comes in, compute a new aggregated value to a 4th topic. 
Each of the 3 topics has components that make up the output (though one 
is required and is guaranteed to be generated before any of the others). 
I would also like the output topic to be compacted. My first doubt is 
whether I should use windowing or not. I want to get low-latency 
throughput. Second, it's not clear to me how I should put the pipeline 
together. Do I start with an empty KTable and stream all 3 topics into 
it using join operators? TIA for any help. This looks like a great 
technology, and we have multiple use cases of event enriching that we'd 
like to do with this technique.

thanks, brian

View raw message