spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From didmar <marin.did...@gmail.com>
Subject Managing permissions when saving as text file
Date Thu, 05 Mar 2015 10:33:35 GMT
Hi,

I'm having a problem involving file permissions on the local filesystem.

On a first machine, I have two different users :
- launcher, which launches my job from an uber jar file
- spark, which runs the master
On a second machine, I have a user spark (same uid/gid as the other) which
runs the worker

Results are written in a shared folder /home/spark/output/ (same path for
both machines) which is owned by spark and has 777 permissions

When I run a job that saves a text file to /home/spark/output/result/,
launcher creates the result folder and subfolders with 775 permissions.
The problem is that the worker, which runs has user spark, cannot write the
results :



I tried to add the sticky bit to /home/spark/output/, but this did not
suffice : it creates a part-00000, starts filling it but then the following
error occurs



Is there a way to force Spark (the driver part I suppose ?) to create all
these folders with 777 permissions instead of 775 ? Or maybe there is
another way ?

Thanks,
Didier



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Managing-permissions-when-saving-as-text-file-tp21928.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message