hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9902) Shell script rewrite
Date Fri, 08 Aug 2014 03:59:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090273#comment-14090273

Allen Wittenauer commented on HADOOP-9902:

bq. It seems that hadoop-functions.sh ends up in sbin/hadoop-functions.sh in the finally binary
assembly but hadoop-config.sh looks for it in the HADOOP_LIBEXEC_DIR. 

Weird!  I've been using tar ball builds and it shows up in HADOOP_LIBEXEC_DIR / libexec as
expected. I wonder what's going on here then. hadoop-dist.xml even specifies libexec and excludes
it from sbin specifically.  From my own fresh build of trunk + this patch:

mbp:hadoop-3.0.0-SNAPSHOT aw$ pwd
mbp:hadoop-3.0.0-SNAPSHOT aw$ find . -name hadoop-functions.sh

So I guess I need some advice here on a way to reproduce it and/or how to fix this one.

bq. hadoop-common-project/hadoop-common/src/main/bin/hadoop's extra quotes

Yup, that's clearly broken.  Probably snuck in while I was playing with the space bug.  That's
an easy fix.

bq.  HADOOPOSTYPE needs to be documents

This is one of those things that I didn't know what to do with it.  I really don't want it
to exist, to be honest, but with HADOOP-8719, we have to do something in the hadoop-env.sh.
Stupid Apple and/or Oracle.  I think in the end we're just kind of screwed and will need to
make it real. :/  I'll change it to HADOOP_OS_TYPE, define it both in hadoop-env.sh and hadoop-functions.sh
so that it can be referenced as a real value.

bq. Are we planning to use *-env.sh for documenting all the variables that one may set?

That was my intent, yes.  I'm looking at it from the perspective of your typical /etc/default/*,
/etc/sysconfig, etc, type file.

bq. Any reason populate_slaves_file function is in hadoop-config.sh and not in hadoop-functions.sh

It was really meant as a utility function only for hadoop-config.sh to simplify the options
listing/parsing code. But there's no reason it can't be moved.  I'll do that as well as make
it hadoop_ to keep the name space clean.  

bq. The following appears to be a sort of no-op if value is not set 

Yup, correct. Another easy fix. (... and clearly bad copypasta from the current code ...)

bq. Any reason not to try harder and see what type -p java returns? 

I think it's too risky.  I'd much rather have an explicit JAVA_HOME than assume that /usr/bin/java
(or whatever happens to get returned) is "good".  'type -p' can also be wildly unpredictable
since it uses the hashed value... this is not necessarily the first one that shows up in the


> Shell script rewrite
> --------------------
>                 Key: HADOOP-9902
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9902
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: scripts
>    Affects Versions: 3.0.0
>            Reporter: Allen Wittenauer
>            Assignee: Allen Wittenauer
>              Labels: releasenotes
>         Attachments: HADOOP-9902-10.patch, HADOOP-9902-11.patch, HADOOP-9902-12.patch,
HADOOP-9902-13-branch-2.patch, HADOOP-9902-13.patch, HADOOP-9902-2.patch, HADOOP-9902-3.patch,
HADOOP-9902-4.patch, HADOOP-9902-5.patch, HADOOP-9902-6.patch, HADOOP-9902-7.patch, HADOOP-9902-8.patch,
HADOOP-9902-9.patch, HADOOP-9902.patch, HADOOP-9902.txt, hadoop-9902-1.patch, more-info.txt
> Umbrella JIRA for shell script rewrite.  See more-info.txt for more details.

This message was sent by Atlassian JIRA

View raw message