spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Theodore Vasiloudis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-5838) Changing SPARK_LOCAL_DIRS option in spark-env.sh does not take effect without daemon restart
Date Thu, 05 Mar 2015 17:25:38 GMT

    [ https://issues.apache.org/jira/browse/SPARK-5838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349131#comment-14349131
] 

Theodore Vasiloudis commented on SPARK-5838:
--------------------------------------------

I had misunderstood the purpose of spark-env then, thanks for clearing this up. Should I close
this as invalid?

> Changing SPARK_LOCAL_DIRS option in spark-env.sh does not take effect without daemon
restart
> --------------------------------------------------------------------------------------------
>
>                 Key: SPARK-5838
>                 URL: https://issues.apache.org/jira/browse/SPARK-5838
>             Project: Spark
>          Issue Type: Bug
>          Components: Deploy, EC2, Spark Submit
>    Affects Versions: 1.1.1
>            Reporter: Theodore Vasiloudis
>            Priority: Minor
>
> This issue has already been mentioned in the mailing list here: http://apache-spark-user-list.1001560.n3.nabble.com/set-spark-local-dir-on-driver-program-doesn-t-take-effect-td11040.html
> The problem usually has to do with Spark creating too many files during shuffles, filling
up the small amount of disk space that most EC2 instances have for root on /mnt2.
> The workaround is to set SPARK_LOCAL_DIRS to a larger volume (e.g. to the /mnt/spark
volume only, removing /mnt2).
> However for these changes to take effect, the daemons need to be restarted with sbin/stop-all
-> sbin/start-all. 
> Even more troubling is the fact that the Web UI-> Environment reports that the spark.local.dir
is set to the new path, but Spark still spills to /mnt2 as well.
> To my knowledge this is not mentioned anywhere in the documentation or any other mailing
list reply except for the one I linked.
> I guess possible solutions are to either ensure the change does take effect so that reality
agrees with what the Web UI is reporting, or include a section on the documentation of EC2
for this kind of problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message