kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Adding topics to KafkaStreams after ingestion has been started?
Date Fri, 02 Dec 2016 18:09:47 GMT
1) There will be once consumer per thread. The number of thread is
defined by the number of instances you start and how many threads you
configure for each instance via StreamConfig parameter
NUM_STREAM_THREADS_CONFIG. Thus, you control this completely by yourself.

Depending on the number to partitions in your topics, each thread will
process one or multiple partitions. As a partition will be processed by
exactly one thread, the overall number of partitions over all you input
topics limits your max number of thread (if you have more threads, those
will just be idle)

2) Thus, there should be no performance issues. Furthermore, if you
create new topic while you application is running -- and if this might
overload you current application -- you can always start new instances
an scale-out you application dynamically -- Kafka Streams is fully elastic.

Have a look here for more details:
http://docs.confluent.io/current/streams/architecture.html


-Matthias

On 12/2/16 6:23 AM, Ali Akhtar wrote:
> That's pretty useful to know - thanks.
> 
> 1) If I listened too foo-.*, and there were 5 foo topics created after
> kafka streaming was running: foo1, foo2, foo3, foo4, foo5, will this create
> 5 consumers / threads / instances, or will it be just 1 instance that
> receives the messages for all of those topics?
> 
> 2) Will this cause issues performance issues if i had a lot of throwaway
> foo topics being created, or will this scale?
> 
> On Fri, Dec 2, 2016 at 7:17 PM, Damian Guy <damian.guy@gmail.com> wrote:
> 
>> Hi Ali,
>>
>> The only way KafkaStreams will process new topics after start is if the
>> original stream was defined with a regular expression, i.e,
>> kafka.stream(Pattern.compile("foo-.*");
>>
>> If any new topics are added after start that match the pattern, then they
>> will also be consumed.
>>
>> Thanks,
>> Damian
>>
>> On Fri, 2 Dec 2016 at 13:13 Ali Akhtar <ali.rac200@gmail.com> wrote:
>>
>>> Heya,
>>>
>>> Normally, you add your topics and their callbacks to a StreamBuilder, and
>>> then call KafkaStreams.start() to start ingesting those topics.
>>>
>>> Is it possible to add a new topic to the StreamBuilder, and start
>> ingesting
>>> that as well, after KafkaStreams.start() has been called?
>>>
>>> Thanks.
>>>
>>
> 


Mime
View raw message