spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Clasen <>
Subject KafkaInputDStream mapping of partitions to tasks
Date Thu, 27 Mar 2014 18:09:16 GMT
I have a simple streaming job that creates a kafka input stream on a topic
with 8 partitions, and does a forEachRDD

The job and tasks are running on mesos, and there are two tasks running, but
only 1 task doing anything.

I also set spark.streaming.concurrentJobs=8  but still there is only 1 task
doing work. I would have expected that each task took a subset of the

Is there a way to make more than one task share the work here?  Are my
expectations off here?

View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message