kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avi Flax <avi.f...@parkassist.com>
Subject [Streams] Threading Frustration
Date Mon, 12 Dec 2016 15:02:34 GMT
Hi all,

I’m running Kafka 0.10.0.1 on Java 8 on Linux — same for brokers and streams nodes.

I’m attempting to scale up a Streams app that does a lot of I/O — in particular, I’m
hoping to isolate each partition into its own thread — and I’m confused:

* I recently created 11 new topics and partitions and modified this app to consume from the
11 new topics

* The app is now consuming from 130 partitions across those 11 topics

* I’ve set the config value num.stream.threads to 130

* In the logs from the startup routine, I can see that Streams is creating 130 threads

* When I use kafka-consumer-groups to see the status of the app, in particular lag, I can
see that it’s only using 20 of those threads to consume from its topics

* This means, of course, that when processing for one partition blocks, it blocks processing
for any other partition assigned to the same thread.


So:

* Why is this happening?

* How can I achieve my goal?


Thank you,
Avi


————
Software Architect @ Park Assist
We’re hiring! http://tech.parkassist.com/jobs/
Mime
View raw message