kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: kafka streams and broadcast topic
Date Mon, 02 Jan 2017 19:05:58 GMT
I think what you're describing could be handled in KStreams by a "global"
KTable. This functionality is currently being discussed/voted on in a KIP
discussion: https://cwiki.apache.org/confluence/pages/viewpa
ge.action?pageId=67633649 The list of interests would be a global KTable
(shared globally across all streams instances) and each instance would
filter/categorize based on that table via a join.

-Ewen

On Fri, Dec 30, 2016 at 3:02 PM, Matt King <kyrrigle@gmail.com> wrote:

> I'd like to have the following:
>
> One large stream of content coming through a topic, with Kafka Stream
> filtering to identify records of interest.   I can see how this would be
> sharded to allow scale out to handle a large stream of content.
>
> I would like to have a 2nd, smaller, topic to define the areas of interest
> that would be used for the filtering.  This topic should be available to
> all the stream processing filters.  When a new filter comes up it should be
> able to recreate its state.   As area of interest definitions change these
> changes should also go out to all the filtering applications.
>
> Can this be done directly with Kafka Streams?  I can get the primary stream
> working with a static set of interests and the filtering works fine.  But
> adding in a second input stream I'm having trouble.  There doesn't seem to
> be a way to have the same topic/partition go to all the applications?
> Alternatively I could imagine broadcasting the interests to multiple
> partitions but don't see how that is done.
>
> Perhaps the area-of-interest topic should be done using a plain old Kafka
> producer, sending it to all partitions?
>
> Am I making sense?
>
> Happy New Year
>
> Matt
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message