hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Burlison (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11505) Various native parts use bswap incorrectly and unportably
Date Tue, 19 Jan 2016 16:45:39 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106978#comment-15106978
] 

Alan Burlison commented on HADOOP-11505:
----------------------------------------

You can't buildhadoop-mapreduce-project or hadoop-mapreduce-client on their own anyway, despite
what it says in BUILDING.txt - or at least I can't:

{noformat}
Where to run Maven from?

  It can be run from any module. The only catch is that if not run from utrunk
  all modules that are not part of the build run must be installed in the local
  Maven cache or available in a Maven repository.
{noformat}

If I try to build from either {{hadoop-mapreduce-project}} or {{hadoop-mapreduce-project/hadoop-mapreduce-client}}
it results in:

{noformat}
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /pool/home/alanbur/bigdata/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskAttemptFinishingMonitor.java:[41,42]
SystemClock() has private access in org.apache.hadoop.yarn.util.SystemClock
[ERROR] /pool/home/alanbur/bigdata/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java:[248,9]
SystemClock() has private access in org.apache.hadoop.yarn.util.SystemClock
[INFO] 2 errors 
{noformat}

The only way I can get it to build is to do a full build or to use {{mvn -pl hadoop-mapreduce-project
-am compile}} from the repo root directory. And if I do that it only takes 33 seconds to do
a clean build versus 19 seconds to do a nothing-has-changed rebuild. That's because stuff
gets needlessly rebuilt even when nothing has changed:

{noformat}
[INFO] Compiling 862 source files to /pool/home/alanbur/bigdata/hadoop/hadoop-common-project/hadoop-common/target/classes
[INFO] Compiling 228 source files to /pool/home/alanbur/bigdata/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/target/classes
{noformat}

I note also that the above incantation already builds Hadoop-Common, so there's no increase
in build time anyway. If there's a problem to be fixed it's the one that causes 3180 files
to be unnecessarily recompiled during a full build even when nothing has changed:

{noformat}
[INFO] Compiling 862 source files to /pool/home/alanbur/bigdata/hadoop/hadoop-common-project/hadoop-common/target/classes
[INFO] Compiling 228 source files to /pool/home/alanbur/bigdata/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/target/classes
[INFO] Compiling 229 source files to /pool/home/alanbur/bigdata/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/target/classes
[INFO] Compiling 346 source files to /pool/home/alanbur/bigdata/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/classes
[INFO] Compiling 78 source files to /pool/home/alanbur/bigdata/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/target/classes
[INFO] Compiling 217 source files to /pool/home/alanbur/bigdata/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/classes
[INFO] Compiling 388 source files to /pool/home/alanbur/bigdata/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/classes
[INFO] Compiling 48 source files to /pool/home/alanbur/bigdata/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/target/classes
[INFO] Compiling 507 source files to /pool/home/alanbur/bigdata/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/classes
[INFO] Compiling 129 source files to /pool/home/alanbur/bigdata/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/target/classes
[INFO] Compiling 48 source files to /pool/home/alanbur/bigdata/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/target/classes
[INFO] Compiling 95 source files to /pool/home/alanbur/bigdata/hadoop/hadoop-tools/hadoop-rumen/target/classes
[INFO] Compiling 5 source files to /pool/home/alanbur/bigdata/hadoop/hadoop-tools/hadoop-extras/target/classes
{noformat}

And that's just the Java files. If the unnecessary regeneration of Makefiles and the consequential
recompilations were prevented then even more time would be saved.

> Various native parts use bswap incorrectly and unportably
> ---------------------------------------------------------
>
>                 Key: HADOOP-11505
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11505
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Alan Burlison
>             Fix For: 3.0.0
>
>         Attachments: HADOOP-11505.001.patch, HADOOP-11505.003.patch, HADOOP-11505.004.patch,
HADOOP-11505.005.patch, HADOOP-11505.006.patch, HADOOP-11505.007.patch
>
>
> hadoop-mapreduce-client-nativetask fails to use x86 optimizations in some cases.  Also,
on some alternate, non-x86, non-ARM architectures the generated code is incorrect.  Thanks
to Steve Loughran and Edward Nevill for finding this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message