spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fawze Abujaber <fawz...@gmail.com>
Subject Re: Unable to see completed application in Spark 2 history web UI
Date Fri, 17 Aug 2018 09:02:25 GMT
Thanks Manu for your response.

I already checked the logs and didn't see anything that can help me
understanding the issue.

The more weird thing, i have a small CI cluster which run on single
NameNode and i see the Spark2 job in the UI, i'm still not sure if it may
related to the NameNode HA, i tried to replace the logdir from NameNode HA
to the activeNameNode like this
http://server:8020/user/spark/spark2historyapplication in the spark2
default conf but the UI still showing the the path with the HA NameNode
event after a restart of Spark2.

The issue become more intersting :)

On Fri, Aug 17, 2018 at 2:01 AM Manu Zhang <owenzhang1990@gmail.com> wrote:

> Hi Fawze,
>
> Sorry but I'm not familiar with CM. Maybe you can look into the logs (or
> turn on DEBUG log).
>
> On Thu, Aug 16, 2018 at 3:05 PM Fawze Abujaber <fawzeaj@gmail.com> wrote:
>
>> Hi Manu,
>>
>> I'm using cloudera manager with single user mode and every process is
>> running with cloudera-scm user, the cloudera-scm is a super user and this
>> is why i was confused how it worked in spark 1.6 and not in spark 2.3
>>
>>
>> On Thu, Aug 16, 2018 at 5:34 AM Manu Zhang <owenzhang1990@gmail.com>
>> wrote:
>>
>>> If you are able to log onto the node where UI has been launched, then
>>> try `ps -aux | grep HistoryServer` and the first column of output should be
>>> the user.
>>>
>>> On Wed, Aug 15, 2018 at 10:26 PM Fawze Abujaber <fawzeaj@gmail.com>
>>> wrote:
>>>
>>>> Thanks Manu, Do you know how i can see which user the UI is running,
>>>> because i'm using cloudera manager and i created a user for cloudera
>>>> manager and called it spark but this didn't solve me issue and here i'm
>>>> trying to find out the user for the spark hisotry UI.
>>>>
>>>> On Wed, Aug 15, 2018 at 5:11 PM Manu Zhang <owenzhang1990@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Fawze,
>>>>>
>>>>> A) The file permission is currently hard coded to 770 (
>>>>> https://github.com/apache/spark/blob/branch-2.3/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L287
>>>>> ).
>>>>> B) I think add all users (including UI) to the group like Spark will
>>>>> do.
>>>>>
>>>>>
>>>>> On Wed, Aug 15, 2018 at 6:38 PM Fawze Abujaber <fawzeaj@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Manu,
>>>>>>
>>>>>> Thanks for your response.
>>>>>>
>>>>>> Yes, i see but still interesting to know how i can see these
>>>>>> applications from the spark history UI.
>>>>>>
>>>>>> How i can know with which user i'm  logged in when i'm navigating
the
>>>>>> spark history UI.
>>>>>>
>>>>>> The Spark process is running with cloudera-scm and the events written
>>>>>> in the spark2history folder at the HDFS written with the user name
who is
>>>>>> running the application and group spark (770 permissions).
>>>>>>
>>>>>> I'm interesting to see if i can force these logs to be written with
>>>>>> 774 or 775 permission or finding another solutions that enable Rnd
or
>>>>>> anyone to be able to investigate his application logs using the UI.
>>>>>>
>>>>>> for example : can i use such spark conf :
>>>>>> spark.eventLog.permissions=755
>>>>>>
>>>>>> The 2 options i see here:
>>>>>>
>>>>>> A) find a way to enforce these logs to be written with other
>>>>>> permissions.
>>>>>>
>>>>>> B) Find the user that the UI running with as creating LDAP groups
and
>>>>>> user that can handle this.
>>>>>>
>>>>>> for example creating group called Spark and create the user that
the
>>>>>> UI running with and add this user to the spark group.
>>>>>> not sure if this option will work as i don't know if these steps
>>>>>> authenticate against the LDAP.
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Take Care
>>>> Fawze Abujaber
>>>>
>>>
>>
>> --
>> Take Care
>> Fawze Abujaber
>>
>

-- 
Take Care
Fawze Abujaber

Mime
View raw message