spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Theodore Vasiloudis (JIRA)" <>
Subject [jira] [Commented] (SPARK-5838) Changing SPARK_LOCAL_DIRS option in does not take effect without daemon restart
Date Thu, 05 Mar 2015 17:25:38 GMT


Theodore Vasiloudis commented on SPARK-5838:

I had misunderstood the purpose of spark-env then, thanks for clearing this up. Should I close
this as invalid?

> Changing SPARK_LOCAL_DIRS option in does not take effect without daemon
> --------------------------------------------------------------------------------------------
>                 Key: SPARK-5838
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Deploy, EC2, Spark Submit
>    Affects Versions: 1.1.1
>            Reporter: Theodore Vasiloudis
>            Priority: Minor
> This issue has already been mentioned in the mailing list here:
> The problem usually has to do with Spark creating too many files during shuffles, filling
up the small amount of disk space that most EC2 instances have for root on /mnt2.
> The workaround is to set SPARK_LOCAL_DIRS to a larger volume (e.g. to the /mnt/spark
volume only, removing /mnt2).
> However for these changes to take effect, the daemons need to be restarted with sbin/stop-all
-> sbin/start-all. 
> Even more troubling is the fact that the Web UI-> Environment reports that the spark.local.dir
is set to the new path, but Spark still spills to /mnt2 as well.
> To my knowledge this is not mentioned anywhere in the documentation or any other mailing
list reply except for the one I linked.
> I guess possible solutions are to either ensure the change does take effect so that reality
agrees with what the Web UI is reporting, or include a section on the documentation of EC2
for this kind of problem.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message