spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Rosen (JIRA)" <>
Subject [jira] [Commented] (SPARK-9499) Possible file handle leak in spilling/sort code
Date Tue, 11 Aug 2015 19:11:45 GMT


Josh Rosen commented on SPARK-9499:

I tried running your attached {{perf_test4.scala}} file under YourKit profiler in order to
instrument file opens / closes and while it looks like the peak number of open files can be
quite large (> 2k files or more) and while parts of the query seem to run slow, it seems
that the number of opened / closed files match up over the course of the query (in other words,
I haven't detected any leaks, but the peak number of open files is large).

/cc [~davies], who is investigating a similar issue.

> Possible file handle leak in spilling/sort code
> -----------------------------------------------
>                 Key: SPARK-9499
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Reynold Xin
>            Assignee: Josh Rosen
>            Priority: Blocker
>         Attachments: open.files.II.txt, perf_test4.scala
> As reported by [~hvanhovell]. See SPARK-8850.
> Hi,
> I am getting a Too many open files error since the unsafe mode is on. The same thing
popped up when playing with unsafe before. The error is below:
> {noformat}
> 15/07/30 23:37:29 WARN TaskSetManager: Lost task 2.0 in stage 33.0 (TID 2423, localhost): /tmp/blockmgr-b3d3e14a-f313-4075-8082-7d97f012e35a/14/temp_shuffle_1cab42fa-dcb1-4114-ae53-1674446f9dac
(Too many open files)
> 	at Method)
> 	at
> 	at<init>(
> 	at
> 	at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.insertAll(
> 	at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:73)
> 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:71)
> 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> 	at
> 	at org.apache.spark.executor.Executor$
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(
> 	at java.util.concurrent.ThreadPoolExecutor$
> 	at
> {noformat}
> I am currently working on local mode (which is probably the cause of the problem) using
the following command line:
> {noformat}
> $SPARK_HOME/bin/spark-shell --master local[*] --driver-memory 14G --driver-library-path
> {noformat}
> The maximum number of files I can open are 1024 (ulimit -n). I have tried to run the
same code with an increased limit, but this didn't work out.
> Dump of all open files after a Too Many Files Open error.
> The command used to make the dump:
> {code}
> lsof -c java > open
> {code}
> The job starts crashing after as soon as I start sorting 10000000 rows for the 9th time
(doing benchmarking). I guess files are left open after every benchmark? Is there a way to
trigger the closing of files?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message