kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrik Kleindl <pklei...@gmail.com>
Subject Re: Case of joining multiple streams/tables
Date Fri, 06 Dec 2019 08:08:52 GMT
Hi
https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup
might
be worth a look.
best regards
Patrik

On Fri, 6 Dec 2019 at 06:44, Sachin Mittal <sjmittal@gmail.com> wrote:

> I was thinking more of a builder api at DSL level.
> Something like this:
> StreamsBuilder.joineBuilder()
>                 .join(kstream1)
>                 .leftJoin(kstream2)
>                 .leftJoin(kstream3)
>                 ....
>                 .joiner((k,v1,v2,v3...) -> ...)
>                 .window()
>                 .build();
>
>
> So when we require to join multiple streams, it can be done using one
> joiner.
> Internally this can be implemented at the processor level.
>
> So not so much of just adding another API call, but something that can make
> code cleaner and more efficient, by using single joiner.
>
> Thanks
> Sachin
>
>
>
> On Thu, Dec 5, 2019 at 2:22 PM Bruno Cadonna <bruno@confluent.io> wrote:
>
> > Hi Sachin,
> >
> > I do not completely understand what you mean with one single
> > operation. Do you mean one call of a method in the DSL or the join is
> > processed on one processor node?
> >
> > If you mean the latter, the joins in the DSL are also not processed on
> > one single processor node.
> >
> > If you mean the former, the DSL does not have a single method call to
> > join multiple streams and it does not necessarily need it to process
> > an n-way join more efficiently, because the DSL is just the way you
> > declare the join. How the join is processed depends on how the
> > topology is build from the DSL code. Having a DSL call specific for a
> > n-way join would merely result in syntactic sugar (which can also make
> > sense).
> >
> > If you have specific requirements that are not fulfilled by the DSL
> > you can use the Processor API to implement your own join.
> >
> > See the following StackOverflow question for more details on joins.
> >
> >
> https://stackoverflow.com/questions/53485632/kafka-streams-implementing-joining-using-the-processor-api
> >
> > Best,
> > Bruno
> >
> > On Thu, Dec 5, 2019 at 7:08 AM Sachin Mittal <sjmittal@gmail.com> wrote:
> > >
> > > Hi,
> > > I have checked the documentation and what I see that we can join two
> > > streams or tables at a given time.
> > >
> > > I have a case where I have multiple streams which I need to join based
> on
> > > common key.
> > >
> > > As of now I am first joining two and the result of that with next and
> so
> > on.
> > >
> > > Is there a way or any case implemented anywhere that joins multiple
> > > streams/tables in a single operation.
> > >
> > > If not then is this something that is pipelined for future releases?
> > > Or does something like this make sense to be part of streams
> > functionality?
> > >
> > > Thanks
> > > Sachin
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message