spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fawze Abujaber <fawz...@gmail.com>
Subject Unable to see completed application in Spark 2 history web UI
Date Wed, 08 Aug 2018 04:56:41 GMT
Hello Community,

I'm using Spark 2.3 and Spark 1.6.0 in my cluster with Cloudera
distribution 5.13.0.

Both are configured to run on Yarn, but i'm unable to see completed
application in Spark2 history server, while in Spark 1.6.0 i did.

1) I checked the HDFS permissions for both folders and both have the same
permissions.

drwxrwxrwt   - cloudera-scm spark          0 2018-08-08 00:46
/user/spark/applicationHistory
drwxrwxrwt   - cloudera-scm spark          0 2018-08-08 00:46
/user/spark/spark2ApplicationHistory

The applications file itself running with permissions 770 in both.

-rwxrwx---   3  fawzea spark     4743751 2018-08-07 23:32
/user/spark/spark2ApplicationHistory/application_1527404701551_672816_1
-rwxrwx---   3  fawzea spark       134315 2018-08-08 00:41
/user/spark/applicationHistory/application_1527404701551_673359_1

2) No error in the Spark2 history server log.

3) Compared the configurations between Spark 1.6 and Spark 2.3 like system
user, enable log, etc ... all looks the same.

4) Once i changed the permissions for the above Spark2 applications to 777,
i was able to see the application in the spark2 history server UI.

Tried to figure out if the 2 Sparks UIs running with different users but
was unable to find it.

Anyone who ran into this issue and solved it?

Thanks in advance.


-- 
Take Care
Fawze Abujaber

Mime
View raw message