kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damian Guy <damian....@gmail.com>
Subject Re: Streams - merging multiple topics
Date Mon, 21 Nov 2016 21:03:55 GMT
Hi Brian,

It sounds like you might want do something like:

KTable inputOne = builder.table("input-one");
KTable inputTwo = builder.table("input-two");
KTable inputThree = builder.table("input-three");
ValueJoiner joiner1 = //...
ValueJoiner joiner2 = //...

inputOne.join(inputTwo, joiner1)
               .join(inputThree, joiner2)

This would result in the join logic being triggered when any record arrives
in any of the input topics. There will be some de-duplication of results
written to the output topic. If you want immediate output then you will
want to set StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG = 0. You should
create and configure the output topic yourself before you run your streams
app (so you can make it compacted etc).


On Mon, 21 Nov 2016 at 12:02 Brian Krahmer <bkrahmer@krahmer.com> wrote:

> Hey guys. I've been banging my head for about 3 days now trying to get a
> streams application working with no luck. I've read through all of the
> docs and examples I can find, and just am not getting it.  I'm using
> 0.10.1 and have worked quite a bit with the high-level consumer and
> publisher. What I would like to do is stream from 3 different topics
> (all keyed by the same uuid, with each having their own value type), and
> when a message comes in, compute a new aggregated value to a 4th topic.
> Each of the 3 topics has components that make up the output (though one
> is required and is guaranteed to be generated before any of the others).
> I would also like the output topic to be compacted. My first doubt is
> whether I should use windowing or not. I want to get low-latency
> throughput. Second, it's not clear to me how I should put the pipeline
> together. Do I start with an empty KTable and stream all 3 topics into
> it using join operators? TIA for any help. This looks like a great
> technology, and we have multiple use cases of event enriching that we'd
> like to do with this technique.
> thanks, brian

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message