hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sangjin Lee <sjl...@gmail.com>
Subject Re: should clients be able to override *-site.xml?
Date Thu, 11 Sep 2014 15:38:43 GMT
Thanks for the clarification. Yes I agree that there are many ways for
users to override *-site.xml regardless of what the hadoop scripts do. It
was more of a question of the interpretation of intent, and I got the
answer. Thanks!

Sangjin

On Wed, Sep 10, 2014 at 4:34 PM, Chris Nauroth <cnauroth@hortonworks.com>
wrote:

> I participated on the HADOOP-9450 code review.  There was some debate about
> the significance of keeping HADOOP_CONF_DIR first.  Ultimately, there was
> agreement that "FIRST" really ought to mean first.  Apparently, other
> Hadoop ecosystem projects have made the same interpretation and implemented
> their scripts accordingly.  Just echoing what Allen said, there were always
> plenty of ways for a user to override *-site.xml too.
>
> Of course, I'm sorry to hear the change caused an issue at your company,
> Sangjin.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
> On Wed, Sep 10, 2014 at 10:55 AM, Allen Wittenauer <aw@altiscale.com>
> wrote:
>
> >
> > On Sep 10, 2014, at 10:14 AM, Sangjin Lee <sjlee@apache.org> wrote:
> >
> > >  if HADOOP_USER_CLASSPATH_FIRST is
> > > set and the user provides his/her version of *-site.xml through
> > > HADOOP_CLASSPATH, the user would end up trumping the hadoop
> > configuration.
> > > And I believe this behavior is preserved after Allen's changes
> > > (HADOOP-9902).
> > >
> >
> >         I'd be surprised if 9902 changed this behavior, especially given
> > that HADOOP_CONF_DIR is added fairly quickly to the CLASSPATH and
> > HADOOP_USER_CLASSPATH is processed last or near last.
> >
> > > Is this an intended behavior? What I'm not sure of is whether we expect
> > the
> > > client to be able to override the site.xml files the hadoop
> configuration
> > > provides. If that is true, then it is working as desired. If not, we'd
> > need
> > > to fix this behavior.
> > >
> > > Thoughts?
> > >
> >
> >         Users have always been able to override the *-site.xml files via
> > java properties, jobconf/etc,  pointing HADOOP_CONF_DIR to somewhere
> else,
> > or providing --conf flags to shell commands.  In other words, this isn't
> > new behavior, but provides yet another option to override the provided
> > settings.
> >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message