spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jungtaek Lim <kabh...@gmail.com>
Subject [DISCUSS] Change default executor log URLs for YARN
Date Thu, 07 Feb 2019 08:45:24 GMT
Hi devs,

Based on the suggestion Tom Graves gave me in SPARK-26792
<https://issues.apache.org/jira/browse/SPARK-26792>, I'd like to hear
voices on changing default executor log URLs for YARN, specifically
removing "stdout" and "stderr" and provide link which shows log file"s".
For example, instead of referring two links below:

http://
<NM_HOST>:<NM_PORT>/node/containerlogs/<CONTAINER_ID>/<USER>/<stdout|stderr>?start=-4096

we just refer only one link below:

http://<NM_HOST>:<NM_PORT>/node/containerlogs/<CONTAINER_ID>/<USER>

I've checked new URL works with redirection on NM to jobhistory, so it
won't break what we currently supported. Going through the actual log file
would require two clicks instead of one click though.

Given it introduces the change on UX I'd like to hear voices on this before
submitting a patch. If we'd rather keep this as it is, I would just open
the chance to apply custom log URL for Spark UI as well.

Thanks in advance!

FYI, below is the rationalization on discussion:

While I worked regarding SPARK-23155
<https://issues.apache.org/jira/browse/SPARK-23155>, I've got some inputs
around linking "log directory" instead of log urls for each "stdout" and
"stderr", because in real case end users would put more files then only
stdout and stderr (like gc logs).

SPARK-23155 provides the way to modify log URL but it's only applied to
SHS, and in Spark UI in running apps it still only shows "stdout" and
"stderr". SPARK-26792 is for applying this to Spark UI as well, but I've
got suggestion to just change the default log URL.

Thanks again,
Jungtaek Lim (HeartSaVioR)

Mime
View raw message