spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fawze Abujaber <fawz...@gmail.com>
Subject Re: Unable to see completed application in Spark 2 history web UI
Date Mon, 13 Aug 2018 08:53:11 GMT
Hi Guys,

Any help here?

On Wed, Aug 8, 2018 at 7:56 AM Fawze Abujaber <fawzeaj@gmail.com> wrote:

> Hello Community,
>
> I'm using Spark 2.3 and Spark 1.6.0 in my cluster with Cloudera
> distribution 5.13.0.
>
> Both are configured to run on Yarn, but i'm unable to see completed
> application in Spark2 history server, while in Spark 1.6.0 i did.
>
> 1) I checked the HDFS permissions for both folders and both have the same
> permissions.
>
> drwxrwxrwt   - cloudera-scm spark          0 2018-08-08 00:46
> /user/spark/applicationHistory
> drwxrwxrwt   - cloudera-scm spark          0 2018-08-08 00:46
> /user/spark/spark2ApplicationHistory
>
> The applications file itself running with permissions 770 in both.
>
> -rwxrwx---   3  fawzea spark     4743751 2018-08-07 23:32
> /user/spark/spark2ApplicationHistory/application_1527404701551_672816_1
> -rwxrwx---   3  fawzea spark       134315 2018-08-08 00:41
> /user/spark/applicationHistory/application_1527404701551_673359_1
>
> 2) No error in the Spark2 history server log.
>
> 3) Compared the configurations between Spark 1.6 and Spark 2.3 like system
> user, enable log, etc ... all looks the same.
>
> 4) Once i changed the permissions for the above Spark2 applications to
> 777, i was able to see the application in the spark2 history server UI.
>
> Tried to figure out if the 2 Sparks UIs running with different users but
> was unable to find it.
>
> Anyone who ran into this issue and solved it?
>
> Thanks in advance.
>
>
> --
> Take Care
> Fawze Abujaber
>


-- 
Take Care
Fawze Abujaber

Mime
View raw message