kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: How many kafka streams app is recommended to run on single machine in production ?
Date Fri, 19 Jan 2018 23:05:47 GMT
Multiple answers:

- a KafkaStreams instance start one *processing* thread by default (you
can configure more processing threads, too)

- internally, KafkaStreams uses two KafkaConsumers and one KafkaProducer
(if you turn on EOS, it uses even more KafkaProducers): a KafkaConsumer
starts a background heartbeat thread and a KafkaProducer starts a
background sender thread => you get 4 threads in total (processing, 2x
heartbeat, sender) -- if you configure two processing threads, you end
up with 8 threads in total, etc)

- there is more than one TCP connection as the consumer and the producer
(and the restore consumer, if you enable StandbyTasks) connect to the
cluster

- it's not possible to share any TPC connections atm (this would require
a mayor rewrite of consumers and producers)

- how many thread you can efficient run depends on your hardward and
workload... monitor you CPU utilization and see how buys your machine is...


Hope this helps

-Matthias

On 1/19/18 9:23 AM, Saloni Vithalani wrote:
> In our architecture, we are assuming to run three jvm processes on one
> machine (approx.) and each jvm machine can host upto 15 kafka-stream apps.
> 
> And if I am not wrong each kafka-stream app spawns one java thread. So,
> this seems like an awkward architecture to have with around 45 kafka-stream
> apps running on a single machine.
> 
> So, I have question in three parts
> 
> 1) Is my understanding correct that each kafka-stream app spawns one java
> thread ? Also, each kafka-stream starts a new tcp connection with
> kafka-broker ?
> 
> 2) Is there a way to share one tcp connection for multiple kafka-streams ?
> 
> 3) Is is difficult(not recommended) to run 45 streams on single machine ?
> The answer to this is definitely NO unless there is a real use case in
> production.
> 
> Regards,
> Saloni Vithalani
> Developer
> Email saloniv@thoughtworks.com
> Telephone +91 8552889571 <8552889571>
> [image: ThoughtWorks]
> <http://www.thoughtworks.com/?utm_campaign=saloni-vithalani-signature&utm_medium=email&utm_source=thoughtworks-email-signature-generator>
> 


Mime
View raw message