kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Boyang Chen <reluctanthero...@gmail.com>
Subject Re: dynamic produce and consume new topics
Date Sat, 13 Jun 2020 15:35:44 GMT
Good question, TBH I'm not an expert on Spring, maybe cross post this
question in the Spring community?

On Sat, Jun 13, 2020 at 7:43 AM AJ Chen <ajchen@web2express.org> wrote:

> Another strategy to handle dynamic topics:
> SInce Spring Kafka MessageListenerContainer holds a list of topics, can we
> simply add a new topic into the existing container at runtime?
> Does the existing listener container object require a manual stop/start
> after its topic list changes?
>
> thanks,
> -aj
>
>
>
> On Fri, Jun 12, 2020 at 3:25 PM AJ Chen <ajchen@web2express.org> wrote:
>
> > I figured out this approach to consume new topics that become available
> at
> > runtime:
> > 1. create ConcurrentMessageListenerContainer for each new topic, call
> > start() to start listening.
> > 2. track all containers in a map, which listen and consume messages from
> > the assigned topics.
> > 3. call container.stop() when shutdown.
> >
> > Is there anything important I should add to make it more robust?
> > Appreciate any suggestions.
> >
> > -aj
> >
> > On Fri, Jun 12, 2020 at 12:55 PM AJ Chen <ajchen@web2express.org> wrote:
> >
> >> Thanks, Boyang.
> >>
> >> Spring''s KafkaTemplate can easily create new topic on the flight
> >> already.
> >>
> >> On consumer side, is there java sample code to replace this annotation?
> >> That will create a new listener for a new topic and group-id on the
> >> fliight.
> >>  @KafkaListener(topics = "test", groupId = "grp0")
> >>
> >> -aj
> >>
> >> On Fri, Jun 12, 2020 at 9:30 AM Boyang Chen <reluctanthero104@gmail.com
> >
> >> wrote:
> >>
> >>> Hey AJ,
> >>>
> >>> you should be able to make the subscribed topics and output topics
> >>> dynamic
> >>> configs for Consumer and Producer. If you need to create new topics on
> >>> runtime, there is some broker side setting you could use to allow topic
> >>> creation based of produced records:
> >>>
> >>>
> https://stackoverflow.com/questions/36441768/how-to-create-topics-in-apache-kafka
> >>>
> >>> On Fri, Jun 12, 2020 at 9:01 AM AJ Chen <canovaj@gmail.com> wrote:
> >>>
> >>> > I'm working on a data streaming system where users will publish to
> >>> > different new topics and subscribe to different new topics without
> >>> > restarting the system. A large number of topics will be created over
> >>> time.
> >>> > Uer will choose different new topics to publish and subscribe
> >>> dynamically
> >>> > at runtime. Can someone share good design or practice to implement
> such
> >>> > dynamic topic handling? Links to sample code will be very helpful.
> >>> >
> >>> > Thanks,
> >>> > aj
> >>> >
> >>>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message