spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jungtaek Lim <>
Subject Re: [DISCUSS] Change default executor log URLs for YARN
Date Sat, 09 Feb 2019 00:38:35 GMT
I'm sure the precondition of changing default behavior is it would help for
many users. If not, we may be better to provide alternative approach for
minority usage. Maybe we tend to go with option 1, or even force them to
use SHS instead (which already enabled the customization of log url).
Option 3 would also cover every cases, if we really don't bother for adding
new option for that.

2019년 2월 9일 (토) 오전 9:31, Ryan Blue <>님이 작성:

> I'm not sure that many people need this, so it is hard to make a decision.
> I'm reluctant to change the current behavior if the result is a new
> papercut to 99% of users and a win for 1%. The suggested change will work
> for 100% of users, so if we don't want a flag then we should go with that.
> But I would certainly want to turn it off in our environment because it
> doesn't provide any value for us and would annoy our users.
> On Fri, Feb 8, 2019 at 4:18 PM Sean Owen <> wrote:
>> Is a flag needed? You know me, I think flags are often failures of
>> design, or disagreement punted to the user. I can understand retaining
>> old behavior under a flag where the behavior change could be
>> problematic for some users or facilitate migration, but this is just a
>> change to some UI links no? the underlying links don't change.
>> On Fri, Feb 8, 2019 at 5:41 PM Ryan Blue <> wrote:
>> >
>> > I suggest using the current behavior as the default and add a flag to
>> implement the behavior you're suggesting: to link to the logs path in YARN
>> instead of directly to stderr and stdout.
>> >
>> > On Fri, Feb 8, 2019 at 3:33 PM Jungtaek Lim <> wrote:
>> >>
>> >> Ryan,
>> >>
>> >> actually I'm not clear about your suggestion. For me three possible
>> options here:
>> >>
>> >> 1. If we want to let users be able to completely rewrite log urls,
>> that's SPARK-26792. For SHS we already addressed it.
>> >> 2. We could let users turning on/off flag option to just get one url
>> or default two stdout/stderr urls.
>> >> 3. We could let users enumerate file names they want to link, and
>> create log links for each file.
>> >>
>> >> Which one do you suggest?
>> >
> --
> Ryan Blue
> Software Engineer
> Netflix

View raw message