spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Same code --works in spark 1.0.2-- but not in spark 1.1.0
Date Thu, 09 Oct 2014 07:37:47 GMT
Can you try decreasing the level of parallelism that you are giving for
those functions? I had this issue when i gave a value > 500 and it was gone
when i dropped it to 200.

Thanks
Best Regards

On Wed, Oct 8, 2014 at 9:28 AM, Andrew Ash <andrew@andrewash.com> wrote:

> Hi Meethu,
>
> I believe you may be hitting a regression in
> https://issues.apache.org/jira/browse/SPARK-3633
>
> If you are able, could you please try running a patched version of Spark
> 1.1.0 that has commit 4fde28c  reverted and see if the errors go away?
> Posting your results on that bug would be useful, especially if the issues
> disappear after the revert.
>
> Thanks!
> Andrew
>
> On Tue, Oct 7, 2014 at 8:27 AM, MEETHU MATHEW <meethu2006@yahoo.co.in>
> wrote:
>
>> Hi all,
>>
>> My code was working fine in spark 1.0.2 ,but after upgrading to 1.1.0,
>> its throwing exceptions and tasks are getting failed.
>>
>> The code contains some map and filter transformations followed by
>> groupByKey (reduceByKey in another code ). What I could find out is that
>> the code works fine until  groupByKey  or reduceByKey  in both
>> versions.But after that the following errors show up in Spark 1.1.0
>>
>> java.io.FileNotFoundException:
>> /tmp/spark-local-20141006173014-4178/35/shuffle_6_0_5161 (Too many open
>> files)
>>         java.io.FileOutputStream.openAppend(Native Method)
>>         java.io.FileOutputStream.<init>(FileOutputStream.java:210)
>>
>> org.apache.spark.storage.DiskBlockObjectWriter.open(BlockObjectWriter.scala:123)
>>
>> org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:192)
>>
>> org.apache.spark.shuffle.hash.HashShuffleWriter$$anonfun$write$1.apply(HashShuffleWriter.scala:67)
>>
>> org.apache.spark.shuffle.hash.HashShuffleWriter$$anonfun$write$1.apply(HashShuffleWriter.scala:65)
>>         scala.collection.Iterator$class.foreach(Iterator.scala:727)
>>         scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>>
>> org.apache.spark.shuffle.hash.HashShuffleWriter.write(HashShuffleWriter.scala:65)
>>
>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
>>
>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>>         org.apache.spark.scheduler.Task.run(Task.scala:54)
>>
>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>         java.lang.Thread.run(Thread.java:701)
>>
>> I cleaned my /tmp directory,changed my local directory to another folder
>> ; but nothing helped.
>>
>> Can anyone say what could  be the reason .?
>>
>> Thanks & Regards,
>> Meethu M
>>
>
>

Mime
View raw message