spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manu Zhang <owenzhang1...@gmail.com>
Subject Re: Unable to see completed application in Spark 2 history web UI
Date Wed, 15 Aug 2018 14:11:30 GMT
Hi Fawze,

A) The file permission is currently hard coded to 770 (
https://github.com/apache/spark/blob/branch-2.3/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L287
).
B) I think add all users (including UI) to the group like Spark will do.


On Wed, Aug 15, 2018 at 6:38 PM Fawze Abujaber <fawzeaj@gmail.com> wrote:

> Hi Manu,
>
> Thanks for your response.
>
> Yes, i see but still interesting to know how i can see these applications
> from the spark history UI.
>
> How i can know with which user i'm  logged in when i'm navigating the
> spark history UI.
>
> The Spark process is running with cloudera-scm and the events written in
> the spark2history folder at the HDFS written with the user name who is
> running the application and group spark (770 permissions).
>
> I'm interesting to see if i can force these logs to be written with 774 or
> 775 permission or finding another solutions that enable Rnd or anyone to be
> able to investigate his application logs using the UI.
>
> for example : can i use such spark conf : spark.eventLog.permissions=755
>
> The 2 options i see here:
>
> A) find a way to enforce these logs to be written with other permissions.
>
> B) Find the user that the UI running with as creating LDAP groups and user
> that can handle this.
>
> for example creating group called Spark and create the user that the UI
> running with and add this user to the spark group.
> not sure if this option will work as i don't know if these steps
> authenticate against the LDAP.
>

Mime
View raw message