spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitay Joffe <>
Subject Spark S3 Performance
Date Thu, 20 Nov 2014 16:54:54 GMT
I have a simple S3 job to read a text file and do a line count.
Specifically I'm doing *sc.textFile("s3n://mybucket/myfile").count*.The
file is about 1.2GB. My setup is standalone spark cluster with 4 workers
each with 2 cores / 16GB ram. I'm using branch-1.2 code built against
hadoop 2.4 (though I'm not actually using HDFS, just straight S3 => Spark).

The whole count is taking on the order of a couple of minutes, which seems
extremely slow.
I've been looking into it and so far have noticed two things, hoping the
community has seen this before and knows what to do...

1) Every executor seems to make an S3 call to read the *entire file* before
making another call to read just it's split. Here's a paste I've cleaned up
to show just one task: I've verified this happens in
every task. It is taking a long time (40-50 seconds), I don't see why it is
doing this?
2) I've tried a few numPartitions parameters. When I make the parameter
anything below 21 it seems to get ignored. Under the hood FileInputFormat
is doing something that always ends up with at least 21 partitions of ~64MB
or so. I've also tried 40, 60, and 100 partitions and have seen that the
performance only gets worse as I increase it beyond 21. I would like to try
8 just to see, but again I don't see how to force it to go below 21.

Thanks for the help,
- Nitay
Founder & CTO

View raw message