spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sanusha <>
Subject parallelism of task executor worker threads during s3 reads
Date Thu, 12 May 2016 07:30:56 GMT

I am using a spark cluster on Amazon (launched using
spark-1.6-prebuilt-with-hadoop-2.6 spark-ec2 script)
to run a scala driver application to read S3 object content in parallel. 

I have tried “s3n://bucket” with sc.textFile as well as set up an RDD with
the S3 keys and then used 
java aws sdk to map it to s3client.getObject(bucket,key) call on each key to
read content. 

In both cases, it gets quite slow as the number of objects increase to even
few thousands.  For example, 
for 8000 objects it took almost 4mins. In contrast, a custom c++ s3 http
client with multiple threads to 
fetch them in parallel on a VM with single cpu, took 1min with 8 threads and
12secs if I use 80 threads.

I observed two issues in both s3n based read, and s3client based calls:
(a) Each slave in spark cluster has 2 vcpus. The job [ref:NOTE] is
partitioned into #tasks, either = #objects,
 or = 2x#nodes. But at the node, it is not always starting 2/more executor
worker threads as I expected.
Instead a single worker thread serially executes the tasks one after the

(b) Each executor worker thread seems to go through each object key
serially, and the thread ends up almost 
always waiting on socket streaming. 

Has anyone seen this or am I doing something wrong? How do I make sure all
the cpus are used? And how to instruct the executors to start multiple
threads to process data in parallel within the partition/node/task? 

NOTE: since sc.textFile as well as map are transformation, I use
resultRDD.count to kick off the actual read and the times are for the count

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message