kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damian Guy <damian....@gmail.com>
Subject Re: Streams - merging multiple topics
Date Mon, 21 Nov 2016 21:03:55 GMT
Hi Brian,

It sounds like you might want do something like:

KTable inputOne = builder.table("input-one");
KTable inputTwo = builder.table("input-two");
KTable inputThree = builder.table("input-three");
ValueJoiner joiner1 = //...
ValueJoiner joiner2 = //...

inputOne.join(inputTwo, joiner1)
               .join(inputThree, joiner2)
               .to(outputTopic)

This would result in the join logic being triggered when any record arrives
in any of the input topics. There will be some de-duplication of results
written to the output topic. If you want immediate output then you will
want to set StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG = 0. You should
create and configure the output topic yourself before you run your streams
app (so you can make it compacted etc).

Thanks,
Damian

On Mon, 21 Nov 2016 at 12:02 Brian Krahmer <bkrahmer@krahmer.com> wrote:

> Hey guys. I've been banging my head for about 3 days now trying to get a
> streams application working with no luck. I've read through all of the
> docs and examples I can find, and just am not getting it.  I'm using
> 0.10.1 and have worked quite a bit with the high-level consumer and
> publisher. What I would like to do is stream from 3 different topics
> (all keyed by the same uuid, with each having their own value type), and
> when a message comes in, compute a new aggregated value to a 4th topic.
> Each of the 3 topics has components that make up the output (though one
> is required and is guaranteed to be generated before any of the others).
> I would also like the output topic to be compacted. My first doubt is
> whether I should use windowing or not. I want to get low-latency
> throughput. Second, it's not clear to me how I should put the pipeline
> together. Do I start with an empty KTable and stream all 3 topics into
> it using join operators? TIA for any help. This looks like a great
> technology, and we have multiple use cases of event enriching that we'd
> like to do with this technique.
>
> thanks, brian
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message