spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Blue <rb...@netflix.com.INVALID>
Subject Re: [DISCUSS] Change default executor log URLs for YARN
Date Thu, 07 Feb 2019 16:33:18 GMT
Jungtaek,

What is shown at the new URL and how would this improve usability?

On Thu, Feb 7, 2019 at 12:45 AM Jungtaek Lim <kabhwan@gmail.com> wrote:

> Hi devs,
>
> Based on the suggestion Tom Graves gave me in SPARK-26792
> <https://issues.apache.org/jira/browse/SPARK-26792>, I'd like to hear
> voices on changing default executor log URLs for YARN, specifically
> removing "stdout" and "stderr" and provide link which shows log file"s".
> For example, instead of referring two links below:
>
> http://
> <NM_HOST>:<NM_PORT>/node/containerlogs/<CONTAINER_ID>/<USER>/<stdout|stderr>?start=-4096
>
> we just refer only one link below:
>
> http://<NM_HOST>:<NM_PORT>/node/containerlogs/<CONTAINER_ID>/<USER>
>
> I've checked new URL works with redirection on NM to jobhistory, so it
> won't break what we currently supported. Going through the actual log file
> would require two clicks instead of one click though.
>
> Given it introduces the change on UX I'd like to hear voices on this
> before submitting a patch. If we'd rather keep this as it is, I would just
> open the chance to apply custom log URL for Spark UI as well.
>
> Thanks in advance!
>
> FYI, below is the rationalization on discussion:
>
> While I worked regarding SPARK-23155
> <https://issues.apache.org/jira/browse/SPARK-23155>, I've got some inputs
> around linking "log directory" instead of log urls for each "stdout" and
> "stderr", because in real case end users would put more files then only
> stdout and stderr (like gc logs).
>
> SPARK-23155 provides the way to modify log URL but it's only applied to
> SHS, and in Spark UI in running apps it still only shows "stdout" and
> "stderr". SPARK-26792 is for applying this to Spark UI as well, but I've
> got suggestion to just change the default log URL.
>
> Thanks again,
> Jungtaek Lim (HeartSaVioR)
>


-- 
Ryan Blue
Software Engineer
Netflix

Mime
View raw message