spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Blue <>
Subject Re: [DISCUSS] Change default executor log URLs for YARN
Date Sat, 09 Feb 2019 00:30:39 GMT
I'm not sure that many people need this, so it is hard to make a decision.
I'm reluctant to change the current behavior if the result is a new
papercut to 99% of users and a win for 1%. The suggested change will work
for 100% of users, so if we don't want a flag then we should go with that.
But I would certainly want to turn it off in our environment because it
doesn't provide any value for us and would annoy our users.

On Fri, Feb 8, 2019 at 4:18 PM Sean Owen <> wrote:

> Is a flag needed? You know me, I think flags are often failures of
> design, or disagreement punted to the user. I can understand retaining
> old behavior under a flag where the behavior change could be
> problematic for some users or facilitate migration, but this is just a
> change to some UI links no? the underlying links don't change.
> On Fri, Feb 8, 2019 at 5:41 PM Ryan Blue <> wrote:
> >
> > I suggest using the current behavior as the default and add a flag to
> implement the behavior you're suggesting: to link to the logs path in YARN
> instead of directly to stderr and stdout.
> >
> > On Fri, Feb 8, 2019 at 3:33 PM Jungtaek Lim <> wrote:
> >>
> >> Ryan,
> >>
> >> actually I'm not clear about your suggestion. For me three possible
> options here:
> >>
> >> 1. If we want to let users be able to completely rewrite log urls,
> that's SPARK-26792. For SHS we already addressed it.
> >> 2. We could let users turning on/off flag option to just get one url or
> default two stdout/stderr urls.
> >> 3. We could let users enumerate file names they want to link, and
> create log links for each file.
> >>
> >> Which one do you suggest?
> >

Ryan Blue
Software Engineer

View raw message