spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Blue <>
Subject Re: [DISCUSS] Change default executor log URLs for YARN
Date Thu, 07 Feb 2019 16:33:18 GMT

What is shown at the new URL and how would this improve usability?

On Thu, Feb 7, 2019 at 12:45 AM Jungtaek Lim <> wrote:

> Hi devs,
> Based on the suggestion Tom Graves gave me in SPARK-26792
> <>, I'd like to hear
> voices on changing default executor log URLs for YARN, specifically
> removing "stdout" and "stderr" and provide link which shows log file"s".
> For example, instead of referring two links below:
> http://
> <NM_HOST>:<NM_PORT>/node/containerlogs/<CONTAINER_ID>/<USER>/<stdout|stderr>?start=-4096
> we just refer only one link below:
> http://<NM_HOST>:<NM_PORT>/node/containerlogs/<CONTAINER_ID>/<USER>
> I've checked new URL works with redirection on NM to jobhistory, so it
> won't break what we currently supported. Going through the actual log file
> would require two clicks instead of one click though.
> Given it introduces the change on UX I'd like to hear voices on this
> before submitting a patch. If we'd rather keep this as it is, I would just
> open the chance to apply custom log URL for Spark UI as well.
> Thanks in advance!
> FYI, below is the rationalization on discussion:
> While I worked regarding SPARK-23155
> <>, I've got some inputs
> around linking "log directory" instead of log urls for each "stdout" and
> "stderr", because in real case end users would put more files then only
> stdout and stderr (like gc logs).
> SPARK-23155 provides the way to modify log URL but it's only applied to
> SHS, and in Spark UI in running apps it still only shows "stdout" and
> "stderr". SPARK-26792 is for applying this to Spark UI as well, but I've
> got suggestion to just change the default log URL.
> Thanks again,
> Jungtaek Lim (HeartSaVioR)

Ryan Blue
Software Engineer

View raw message