kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From chintan mavawala <chintan25...@gmail.com>
Subject Re: Dynamic topic subscription without using pattern
Date Sun, 07 Jan 2018 03:37:41 GMT
Discovery of new topic is not a concern. Clients who creates a new topic
sends my process the topic name. In simpler words, my process performs some
operations which any client might be wanting. So they create a Kafka topic
and send me the name of the topic for me to subscribe.

My process needs start consuming from this topic too in addition to already
subscribed topics.

I hope it makes sense.

Regards,
Chintan

On 06-Jan-2018 11:30 PM, "Skip Montanaro" <skip.montanaro@gmail.com> wrote:

I'm a bit confused. How do you discover that new topics have been created?
If you discover a new topic somehow, how do you know it's one you are
interested in?

I suppose you could have a consumer which subscribes to all topics using
pattern=".", then have it just publish new topic names to a fixed "new
topics" topic. You could subscribe to that. Still, with no pattern to the
name you'd need some external information (database query?) to decide if
you wanted to subscribe to it.

Skip

On Jan 6, 2018 10:43 AM, "chintan mavawala" <chintan25487@gmail.com> wrote:

> In theory pattern "." would work. But that would mean subscribing on all
> topics which I don't want.
>
> I share same pain in using pattern which is why I would like to know if it
> is good practice to subscribe to topics after kafka consumer has started.
>
> Regards,
> Chintan
>
> On 06-Jan-2018 5:47 PM, "Skip Montanaro" <skip.montanaro@gmail.com> wrote:
>
> In theory, wouldn't
>
> consumer.subscribe(pattern=".")
>
> work? I say "in theory" because my experience with subscribing by pattern
> hasn't been great. I suspect my mental model of how it's implemented isn't
> a close approximation of reality.
>
> Skip
>
> On Jan 6, 2018 4:07 AM, "chintan mavawala" <chintan25487@gmail.com> wrote:
>
> I want to subscribe to all topics as they are created but topic names does
> not have any pattern. Consumer starts with subscription to let's say 3
> topics and add more topics to exisiting kafka consumer as they are
created.
>
> Regards,
> Chintan
>
> On 04-Jan-2018 11:02 PM, "Jordan Pilat" <jrpilat@gmail.com> wrote:
>
> > Did you want to subscribe to all topics as they are created, or to
> exclude
> > some of them?
> > If the latter, is there a pattern in the names of the topics you wish to
> > exclude?
> >
> > - Jordan Pilat
> >
> > On 2018-01-03 23:39, chintan mavawala <chintan25487@gmail.com> wrote:
> > > Hi,
> > >
> > > We have a use case where we need to start kafka consumer with a fixed
> > list
> > > of kafka topics and add more topics on the fly. Since there is no
> pattern
> > > in names of topics, using pattern for dynamic subscriptions of topics
> is
> > > not feasible.
> > >
> > > Is it a good practice to subscribe to topics on given kafka consumer
on
> > fly
> > > using after starting consumer?
> > >
> > > I understand that KafkaConsumer is not thread safe. Any ill effects
due
> > to
> > > multi threaded env?
> > >
> > > Regards,
> > > Chintan
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message