spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jacky Li (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-4787) Resource unreleased during failure in SparkContext initialization
Date Mon, 08 Dec 2014 16:33:13 GMT
Jacky Li created SPARK-4787:
-------------------------------

             Summary: Resource unreleased during failure in SparkContext initialization
                 Key: SPARK-4787
                 URL: https://issues.apache.org/jira/browse/SPARK-4787
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.1.0
            Reporter: Jacky Li
             Fix For: 1.3.0


When client creates a SparkContext, currently there are many val to initialize during object
initialization. But when there is failure initializing these val, like throwing an exception,
the resources in this SparkContext is not released properly. 
For example, SparkUI object is created and bind to the HTTP server during initialization using
{{ui.foreach(_.bind())}}
but if anything goes wrong after this code (say throwing an exception when creating DAGScheduler),
the SparkUI server is not stopped, thus the port bind will fail again in the client when creating
another SparkContext. So basically this leads to a situation that the client can not create
another SparkContext in the same process, which I think it is not reasonable.

So, I suggest to refactor the SparkContext code to release resource when there is failure
during in initialization.
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message