kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AJ Chen <ajc...@web2express.org>
Subject Re: dynamic produce and consume new topics
Date Sat, 13 Jun 2020 14:42:41 GMT
Another strategy to handle dynamic topics:
SInce Spring Kafka MessageListenerContainer holds a list of topics, can we
simply add a new topic into the existing container at runtime?
Does the existing listener container object require a manual stop/start
after its topic list changes?

thanks,
-aj



On Fri, Jun 12, 2020 at 3:25 PM AJ Chen <ajchen@web2express.org> wrote:

> I figured out this approach to consume new topics that become available at
> runtime:
> 1. create ConcurrentMessageListenerContainer for each new topic, call
> start() to start listening.
> 2. track all containers in a map, which listen and consume messages from
> the assigned topics.
> 3. call container.stop() when shutdown.
>
> Is there anything important I should add to make it more robust?
> Appreciate any suggestions.
>
> -aj
>
> On Fri, Jun 12, 2020 at 12:55 PM AJ Chen <ajchen@web2express.org> wrote:
>
>> Thanks, Boyang.
>>
>> Spring''s KafkaTemplate can easily create new topic on the flight
>> already.
>>
>> On consumer side, is there java sample code to replace this annotation?
>> That will create a new listener for a new topic and group-id on the
>> fliight.
>>  @KafkaListener(topics = "test", groupId = "grp0")
>>
>> -aj
>>
>> On Fri, Jun 12, 2020 at 9:30 AM Boyang Chen <reluctanthero104@gmail.com>
>> wrote:
>>
>>> Hey AJ,
>>>
>>> you should be able to make the subscribed topics and output topics
>>> dynamic
>>> configs for Consumer and Producer. If you need to create new topics on
>>> runtime, there is some broker side setting you could use to allow topic
>>> creation based of produced records:
>>>
>>> https://stackoverflow.com/questions/36441768/how-to-create-topics-in-apache-kafka
>>>
>>> On Fri, Jun 12, 2020 at 9:01 AM AJ Chen <canovaj@gmail.com> wrote:
>>>
>>> > I'm working on a data streaming system where users will publish to
>>> > different new topics and subscribe to different new topics without
>>> > restarting the system. A large number of topics will be created over
>>> time.
>>> > Uer will choose different new topics to publish and subscribe
>>> dynamically
>>> > at runtime. Can someone share good design or practice to implement such
>>> > dynamic topic handling? Links to sample code will be very helpful.
>>> >
>>> > Thanks,
>>> > aj
>>> >
>>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message