spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manu Zhang <owenzhang1...@gmail.com>
Subject Re: Unable to see completed application in Spark 2 history web UI
Date Wed, 15 Aug 2018 09:39:32 GMT
Hi Fawze,

In Spark 2.3, HistoryServer will check for file permissions when reading
event logs written by your applications. (Please check
https://issues.apache.org/jira/browse/SPARK-20172). With file permissions
of 770, HistoryServer is not permitted to read the event log. That's why
you were able to see applications once changing file permissions to 777.

Regards,
Manu Zhang

On Mon, Aug 13, 2018 at 4:53 PM Fawze Abujaber <fawzeaj@gmail.com> wrote:

> Hi Guys,
>
> Any help here?
>
> On Wed, Aug 8, 2018 at 7:56 AM Fawze Abujaber <fawzeaj@gmail.com> wrote:
>
>> Hello Community,
>>
>> I'm using Spark 2.3 and Spark 1.6.0 in my cluster with Cloudera
>> distribution 5.13.0.
>>
>> Both are configured to run on Yarn, but i'm unable to see completed
>> application in Spark2 history server, while in Spark 1.6.0 i did.
>>
>> 1) I checked the HDFS permissions for both folders and both have the same
>> permissions.
>>
>> drwxrwxrwt   - cloudera-scm spark          0 2018-08-08 00:46
>> /user/spark/applicationHistory
>> drwxrwxrwt   - cloudera-scm spark          0 2018-08-08 00:46
>> /user/spark/spark2ApplicationHistory
>>
>> The applications file itself running with permissions 770 in both.
>>
>> -rwxrwx---   3  fawzea spark     4743751 2018-08-07 23:32
>> /user/spark/spark2ApplicationHistory/application_1527404701551_672816_1
>> -rwxrwx---   3  fawzea spark       134315 2018-08-08 00:41
>> /user/spark/applicationHistory/application_1527404701551_673359_1
>>
>> 2) No error in the Spark2 history server log.
>>
>> 3) Compared the configurations between Spark 1.6 and Spark 2.3 like
>> system user, enable log, etc ... all looks the same.
>>
>> 4) Once i changed the permissions for the above Spark2 applications to
>> 777, i was able to see the application in the spark2 history server UI.
>>
>> Tried to figure out if the 2 Sparks UIs running with different users but
>> was unable to find it.
>>
>> Anyone who ran into this issue and solved it?
>>
>> Thanks in advance.
>>
>>
>> --
>> Take Care
>> Fawze Abujaber
>>
>
>
> --
> Take Care
> Fawze Abujaber
>

Mime
View raw message