spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From didmar <>
Subject Managing permissions when saving as text file
Date Thu, 05 Mar 2015 10:33:35 GMT

I'm having a problem involving file permissions on the local filesystem.

On a first machine, I have two different users :
- launcher, which launches my job from an uber jar file
- spark, which runs the master
On a second machine, I have a user spark (same uid/gid as the other) which
runs the worker

Results are written in a shared folder /home/spark/output/ (same path for
both machines) which is owned by spark and has 777 permissions

When I run a job that saves a text file to /home/spark/output/result/,
launcher creates the result folder and subfolders with 775 permissions.
The problem is that the worker, which runs has user spark, cannot write the
results :

I tried to add the sticky bit to /home/spark/output/, but this did not
suffice : it creates a part-00000, starts filling it but then the following
error occurs

Is there a way to force Spark (the driver part I suppose ?) to create all
these folders with 777 permissions instead of 775 ? Or maybe there is
another way ?


View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message