kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marko Bonaći <marko.bon...@sematext.com>
Subject Re: Publish to 1 topic, consume from N
Date Thu, 15 Sep 2016 17:49:32 GMT
1. You can create N topics
2. You control from producer where each message goes
3. You have consumer that fetches from M different topics:
https://kafka.apache.org/0100/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#subscribe(java.util.Collection)

Isn't this architecture flexible enough for any type of use case? What do
you think cannot be achieved?

Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>

On Thu, Sep 15, 2016 at 11:01 AM, Ali Akhtar <ali.rac200@gmail.com> wrote:

> It sounds like you can implement the 'mapping service'  component yourself
> using Kafka.
>
> Have all of your messages go to one kafka topic. Have one consumer group
> listening to this 'everything goes here' topic. This consumer group acts as
> your mapping service. It looks at each message, and based on your rules, it
> sends that message to a different topic for those specific rules.
>
> Then you have your consumers listening to the specific topics that they
> need to. Your mapping service does the job of redirecting messages from the
> 'everything' topic to the specific topics based on your rules.
>
> On Thu, Sep 15, 2016 at 1:43 PM, Luiz Cordeiro <
> Luiz.Cordeiro@mobilityhouse.com> wrote:
>
> > Hello,
> >
> > We’re considering migrating an AMQ-based platform to Kafka. However our
> > application logic needs an AMQ feature called Dynamic Binding, that is,
> on
> > AMQ one publishes messages to an Exchange, which can be dynamically
> > configured to deliver a copy of the message to several queues, based on
> > binding rules. So when a new client comes alive, it may create its
> binding
> > rules to specify a set of topics to listen to, and receive all the
> messages
> > from these topics on a private queue.
> >
> > I understand that Kafka neither provides this nor will, as it is not its
> > objective, but I was wondering if there’s another component, an overlay
> to
> > Kafka, that could provide this feature while using Kafka behind the
> scenes
> > for the persistence, something like this:
> >
> > Publisher --> Mapping Service --> Kafka <-- Consumers
> >                      ^                          |
> >                      |       Binding rules      |
> >                      \--------------------------/
> >
> > Are you aware of such a component? Otherwise, how would you solve this
> > issue of publish to 1 place and have it replicated on N topics.
> >
> > Best Regards,
> > Luiz
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message