kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Akhtar <ali.rac...@gmail.com>
Subject Re: Publish to 1 topic, consume from N
Date Thu, 15 Sep 2016 09:01:02 GMT
It sounds like you can implement the 'mapping service'  component yourself
using Kafka.

Have all of your messages go to one kafka topic. Have one consumer group
listening to this 'everything goes here' topic. This consumer group acts as
your mapping service. It looks at each message, and based on your rules, it
sends that message to a different topic for those specific rules.

Then you have your consumers listening to the specific topics that they
need to. Your mapping service does the job of redirecting messages from the
'everything' topic to the specific topics based on your rules.

On Thu, Sep 15, 2016 at 1:43 PM, Luiz Cordeiro <
Luiz.Cordeiro@mobilityhouse.com> wrote:

> Hello,
> We’re considering migrating an AMQ-based platform to Kafka. However our
> application logic needs an AMQ feature called Dynamic Binding, that is, on
> AMQ one publishes messages to an Exchange, which can be dynamically
> configured to deliver a copy of the message to several queues, based on
> binding rules. So when a new client comes alive, it may create its binding
> rules to specify a set of topics to listen to, and receive all the messages
> from these topics on a private queue.
> I understand that Kafka neither provides this nor will, as it is not its
> objective, but I was wondering if there’s another component, an overlay to
> Kafka, that could provide this feature while using Kafka behind the scenes
> for the persistence, something like this:
> Publisher --> Mapping Service --> Kafka <-- Consumers
>                      ^                          |
>                      |       Binding rules      |
>                      \--------------------------/
> Are you aware of such a component? Otherwise, how would you solve this
> issue of publish to 1 place and have it replicated on N topics.
> Best Regards,
> Luiz

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message