hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@hortonworks.com>
Subject Notes for working on mapreduce trunk after the MR-279 merge.
Date Thu, 18 Aug 2011 12:49:35 GMT
MR-279 branch is merged into mapreduce trunk and this changes things a
bit for developing on mapreduce.

You can get all the help that is needed from the INSTALL file at
Reproducing some of those contents here for the short-term lookup.

Checking out source code
svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce

Directory structure

  - hadoop-mapreduce ( was mapreduce before)

trunk/hadoop-mapreduce - Classic code. JT/TT reside here
 - build.xml
 - src

trunk/hadoop-mapreduce/ - New code related to yarn reside here.
 - assembly
 - pom.xml
 - hadoop-mr-client
 - hadoop-yarn - Yarn APIs, libraries, and server code
   -- hadoop-yarn-api
   -- hadoop-yarn-common
   -- hadoop-yarn-server - Server code, ResourceManager, NodeManager,
server libraries and tests.
      --- hadoop-yarn-server-common
      --- hadoop-yarn-server-nodemanager
      --- hadoop-yarn-server-resourcemanager
      --- hadoop-yarn-server-tests
 - hadoop-mr-client - MapReduce server and client code
   -- hadoop-mapreduce-client-app
   -- hadoop-mapreduce-client-core
   -- hadoop-mapreduce-client-jobclient
   -- hadoop-mapreduce-client-common
   -- hadoop-mapreduce-client-hs
   -- hadoop-mapreduce-client-shuffle

Building yarn code and install into the local maven cache.
 - mvn clean install
 - In case you want to skip the tests run: mvn clean install -DskipTests

Building classic code once yarn code is built.
 - ant veryclean jar jar-test  -Dresolvers=internal

 1) For hacking on the new yarn+MR code in eclipse, you should run
"mvn eclipse:eclipse" and then import the checked out source root as a
maven project.
 2) For developing on classic JT/TT code, running "ant eclipse" and
importing as java project should continue to work.

Hope that helps. If you run into issues, please send an email or
create a JIRA issue.


View raw message