spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ankur Srivastava <ankur.srivast...@gmail.com>
Subject Re: Spark with Cassandra - Shuffle opening to many files
Date Wed, 07 Jan 2015 21:56:32 GMT
Thank you Cody!!

I am going to try with the two settings you have mentioned.

We are currently running with Spark standalone cluster manager.

Thanks
Ankur

On Wed, Jan 7, 2015 at 1:20 PM, Cody Koeninger <cody@koeninger.org> wrote:

> General ideas regarding too many open files:
>
> Make sure ulimit is actually being set, especially if you're on mesos
> (because of https://issues.apache.org/jira/browse/MESOS-123 )  Find the
> pid of the executor process, and cat /proc/<pid>/limits
>
> set spark.shuffle.consolidateFiles = true
>
> try spark.shuffle.manager = sort
>
>
> On Wed, Jan 7, 2015 at 3:06 PM, Ankur Srivastava <
> ankur.srivastava@gmail.com> wrote:
>
>> Hello,
>>
>> We are currently running our data pipeline on spark which uses Cassandra
>> as the data source.
>>
>> We are currently facing issue with the step where we create an rdd on
>> data in cassandra table and then try to run "flatMapToPair" to transform
>> the data but we are running into "Too many open files". I have already
>> increased the file limits on all the worker and master node by changing the
>> file /etc/system/limits.conf to 65K but that did not help.
>>
>> Is there some setting so that we can restrict shuffle?
>>
>> Also when we use the log4j.properties in conf folder these logs are not
>> getting emitted.
>>
>> Exception in thread "main" org.apache.spark.SparkException: Job aborted
>> due to stage failure: Task 20 in stage 1.0 failed 4 times, most recent
>> failure: Lost task 20.3 in stage 1.0 (TID 51,
>> ip-10-87-36-147.us-west-2.aws.neustar.com):
>> java.io.FileNotFoundException:
>> /tmp/spark-local-20150107203209-9333/2f/shuffle_0_20_1017 (Too many open
>> files)
>>
>>         java.io.FileOutputStream.open(Native Method)
>>
>>         java.io.FileOutputStream.<init>(FileOutputStream.java:221)
>>
>>
>> org.apache.spark.storage.DiskBlockObjectWriter.open(BlockObjectWriter.scala:123)
>>
>>
>> org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:192)
>>
>>
>> org.apache.spark.shuffle.hash.HashShuffleWriter$$anonfun$write$1.apply(HashShuffleWriter.scala:67)
>>
>>
>> org.apache.spark.shuffle.hash.HashShuffleWriter$$anonfun$write$1.apply(HashShuffleWriter.scala:65)
>>
>>         scala.collection.Iterator$class.foreach(Iterator.scala:727)
>>
>>         scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>>
>>
>> org.apache.spark.shuffle.hash.HashShuffleWriter.write(HashShuffleWriter.scala:65)
>>
>>
>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
>>
>>
>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>>
>>         org.apache.spark.scheduler.Task.run(Task.scala:54)
>>
>>
>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
>>
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>
>>         java.lang.Thread.run(Thread.java:745)
>>
>>
>> Thanks & Regards
>> Ankur
>>
>
>

Mime
View raw message