kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt King <kyrri...@gmail.com>
Subject kafka streams and broadcast topic
Date Fri, 30 Dec 2016 23:02:19 GMT
I'd like to have the following:

One large stream of content coming through a topic, with Kafka Stream
filtering to identify records of interest.   I can see how this would be
sharded to allow scale out to handle a large stream of content.

I would like to have a 2nd, smaller, topic to define the areas of interest
that would be used for the filtering.  This topic should be available to
all the stream processing filters.  When a new filter comes up it should be
able to recreate its state.   As area of interest definitions change these
changes should also go out to all the filtering applications.

Can this be done directly with Kafka Streams?  I can get the primary stream
working with a static set of interests and the filtering works fine.  But
adding in a second input stream I'm having trouble.  There doesn't seem to
be a way to have the same topic/partition go to all the applications?
Alternatively I could imagine broadcasting the interests to multiple
partitions but don't see how that is done.

Perhaps the area-of-interest topic should be done using a plain old Kafka
producer, sending it to all partitions?

Am I making sense?

Happy New Year

Matt

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message