spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fawze Abujaber <>
Subject Re: Unable to see completed application in Spark 2 history web UI
Date Wed, 15 Aug 2018 10:38:08 GMT
Hi Manu,

Thanks for your response.

Yes, i see but still interesting to know how i can see these applications
from the spark history UI.

How i can know with which user i'm  logged in when i'm navigating the spark
history UI.

The Spark process is running with cloudera-scm and the events written in
the spark2history folder at the HDFS written with the user name who is
running the application and group spark (770 permissions).

I'm interesting to see if i can force these logs to be written with 774 or
775 permission or finding another solutions that enable Rnd or anyone to be
able to investigate his application logs using the UI.

for example : can i use such spark conf : spark.eventLog.permissions=755

The 2 options i see here:

A) find a way to enforce these logs to be written with other permissions.

B) Find the user that the UI running with as creating LDAP groups and user
that can handle this.

for example creating group called Spark and create the user that the UI
running with and add this user to the spark group.
not sure if this option will work as i don't know if these steps
authenticate against the LDAP.

View raw message