hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-6941) The default setting doesn't work for MapReduce job
Date Wed, 16 Aug 2017 20:25:00 GMT
Junping Du created MAPREDUCE-6941:
-------------------------------------

             Summary: The default setting doesn't work for MapReduce job
                 Key: MAPREDUCE-6941
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6941
             Project: Hadoop Map/Reduce
          Issue Type: Bug
    Affects Versions: 3.0.0-beta1
            Reporter: Junping Du
            Priority: Blocker


On the deployment of hadoop 3 cluster (based on current trunk branch) with default settings,
the MR job will get failed as following exceptions:
{noformat}
2017-08-16 13:00:03,846 INFO mapreduce.Job: Job job_1502913552390_0001 running in uber mode
: false
2017-08-16 13:00:03,847 INFO mapreduce.Job:  map 0% reduce 0%
2017-08-16 13:00:03,864 INFO mapreduce.Job: Job job_1502913552390_0001 failed with state FAILED
due to: Application application_1502913552390_0001 failed 2 times due to AM Container for
appattempt_1502913552390_0001_000002 exited with  exitCode: 1
Failing this attempt.Diagnostics: [2017-08-16 13:00:02.963]Exception from container-launch.
Container id: container_1502913552390_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:994)
	at org.apache.hadoop.util.Shell.run(Shell.java:887)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1212)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:295)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.launchContainer(ContainerLaunch.java:455)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:275)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:90)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

{noformat}

This is because mapreduce related jar are not added into yarn setup by default. To make MR
job run successful, we need to add following configurations to yarn-site.xml now:
{noformat}
    <property>
      <name>yarn.application.classpath</name>
      <value>
        ...
        <HADOOP_HOME>/share/hadoop/mapreduce/*,
        <HADOOP_HOME>/share/hadoop/mapreduce/lib/*
        ...
      </value>
{noformat}
But this config is not necessary for previous version of Hadoop. We should fix this issue
before beta release otherwise it will be a regression for configuration changes.

This could be more like a YARN issue (if so, we should move), depends on how we fix it finally.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org


Mime
View raw message