hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-9271) revert Python build scripts from branch-trunk-win
Date Fri, 01 Feb 2013 07:59:13 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-9271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Nauroth updated HADOOP-9271:

    Attachment: HADOOP-9271-branch-trunk-win.1.patch

I'm attaching a patch to remove the Python scripting.  I tested the build on Mac, Windows,
and Ubuntu, including native support and bundling Snappy on Ubuntu.

For the most part, this is just a revert back to the trunk version of each of these pom.xml
files.  There are a few differences for Windows compatibility though:

-                      ROOT=`cd ${basedir}/..;pwd`
+                      ROOT=`cd ../..;pwd`

This command was not working on Windows because it would generate a mix of \ and / in the
destination path.  (i.e. cd C:\foo\bar/..)  Switching to a relative path and using only /
as the path delimiter worked cross-platform.

-                      run tar czf hadoop-${project.version}.tar.gz hadoop-${project.version}
+                      run tar cf hadoop-${project.version}.tar hadoop-${project.version}
+                      run gzip hadoop-${project.version}.tar

The 'z' flag for compression causes tar to fork a separate process for gzip.  GnuWin32 tar
has a limitation in that fork was never implemented, so this would fail on Windows with "Cannot
fork: Function not implemented".  Splitting this into separate tar and gzip commands works

Another option here would have been to control the pipeline explicitly using a shell pipeline
(tar | gzip), but the "run" helper function used here isn't compatible with passing a command
that has a pipe.

-                      if [ $? = 1 ]; then
+                      if [ $? != 0 ]; then

On Windows, failure of the prior command might result in some non-zero exit code other than
1, so I changed this to check for any non-zero exit code.

-                      tar xzf ${basedir}/downloads/apache-tomcat-${tomcat.version}.tar.gz
+                      gzip -cd ../../downloads/apache-tomcat-${tomcat.version}.tar.gz | tar
xf -

See explanation above about tar failure to fork on Windows.  This one was solveable using
gzip | tar.

+                      BIN_DIR="${BUILD_DIR}/bin"
+                      if [ -d $${BIN_DIR} ] ; then
+                        TARGET_BIN_DIR="${BUILD_DIR}/${project.artifactId}-${project.version}/bin"
+                        mkdir -p $${TARGET_BIN_DIR}
+                        cd $${BIN_DIR}
+                        $$TAR * | (cd $${TARGET_BIN_DIR}/; $$UNTAR)
+                      fi

This is required to bundle the native Windows components (winutils.exe and hadoop.dll) from
the bin directory.

> revert Python build scripts from branch-trunk-win
> -------------------------------------------------
>                 Key: HADOOP-9271
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9271
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: build
>    Affects Versions: trunk-win
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HADOOP-9271-branch-trunk-win.1.patch
> On branch-trunk-win, some work was committed to convert custom build scripting to Python
as a cross-platform solution to support builds on both Linux and Windows.  The community has
since rejected this solution.  This issue tracks removal of the Python build code from branch-trunk-win.
> This will restore branch-trunk-win to the same state as trunk, which uses shell for custom
build scripting.  This means that developers working on branch-trunk-win will once again need
sh.exe on the PATH to build the distribution.  This will be a temporary state.  A subsequent
jira will add build scripts specific to Windows, using a language more common in a Windows
development environment.  The community has approved minimal use of multiple platform-specific
scripting languages.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message