In worker machines, you will find $SPARK_HOME/work/

that’s worker dir

Nan Zhu

On Tuesday, January 7, 2014 at 7:29 AM, Archit Thakur wrote:

What do you mean by worker dir?

On Tue, Jan 7, 2014 at 11:43 AM, Nan Zhu <> wrote:
Hi, all

I’m trying to change my worker dir to a mounted disk with larger space

but I found that no document telling me how to do this, 

I have to check the source code and found the following lines

 if (System.getenv("SPARK_WORKER_DIR") != null) {
    workDir = System.getenv("SPARK_WORKER_DIR")

is it forgotten to be documented?


Nan Zhu