spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Assigned] (SPARK-26792) Apply custom log URL to Spark UI
Date Thu, 14 Feb 2019 14:03:00 GMT


Apache Spark reassigned SPARK-26792:

    Assignee:     (was: Apache Spark)

> Apply custom log URL to Spark UI
> --------------------------------
>                 Key: SPARK-26792
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Web UI
>    Affects Versions: 3.0.0
>            Reporter: Jungtaek Lim
>            Priority: Major
> SPARK-23155 enables SHS to set up custom log URLs for incompleted / completed apps.
> While getting reviews from SPARK-23155, I've got two comments which applying custom log
URLs to UI would help achieving it. Quoting these comments here:
> {quote}
> Sorry I haven't had time to look through all the code so this might be a separate jira,
but one thing I thought of here is it would be really nice not to have specifically stderr/stdout.
users can specify any and some tools like oozie by default end up using hadoop
log4j rather then spark log4j, so files aren't necessarily the same. Also users can put in
other logs files so it would be nice to have links to those from the UI. It seems simpler
if we just had a link to the directory and it read the files within there. Other things in
Hadoop do it this way, but I'm not sure if that works well for other resource managers, any
thoughts on that? As long as this doesn't prevent the above I can file a separate jira for
> {quote}
> {quote}
> Hi Tom, +1: singling out stdout and stderr is definitely an annoyance. We
> typically configure Spark jobs to write the GC log and dump heap on OOM
> using <LOG_DIR>,  and/or we use the rolling file appender to deal with
> large logs during debugging. So linking the YARN container log overview
> page would make much more sense for us. We work it around with a custom
> submit process that logs all important URLs on the submit side log.
> {quote}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message