spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <>
Subject HDFS user permissions
Date Thu, 19 Sep 2013 14:14:18 GMT
hello all,
currently spark runs tasks as the user that runs the spark worker daemon.
at least, this is what i observer with standalone spark.

as a result spark does not respect user permissions on HDFS.

i guess this could be fixed by running the tasks as a different user, or
even just by using hadoop's proxy mechanism to access HDFS as that user.

are there plans to fix this? is there a JIRA tickets for this? i couldn't
find it but would like to track this issue.

and what about secure HDFS (kerberos)?

thanks! koert

View raw message