hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "NING DING (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-6590) Update MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV value for loading correct navite lib when using ubertask feature
Date Sun, 27 Dec 2015 01:25:49 GMT

             Summary: Update MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV value for loading correct
navite lib when using ubertask feature
                 Key: MAPREDUCE-6590
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6590
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: mr-am
    Affects Versions: 2.7.1
            Reporter: NING DING
            Assignee: NING DING

By default, the MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set.
If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not configured to
set LD_LIBRARY_PATH, MR AM will fail to load the native library, then you can find the error
message as below.
org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform...
using builtin-java classes where applicable.
I found the patch in https://issues.apache.org/jira/browse/MAPREDUCE-5799 update YARNRunner
code and use MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV as the default value. The code is as
            MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV), conf);
In fact the parameters yarn.app.mapreduce.am.env and yarn.app.mapreduce.am.admin.user.env
are resolved in YARNRunner running on hadoop client host.
But their values, e.g. LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native, will be used on hadoop
cluster hosts.
If the hadoop client host installs hadoop on a different path from that on hadoop cluster
the $HADOOP_COMMON_HOME/lib/native will be resolved to a path which is not existed on hadoop
cluster hosts.
Then MRAppMaster running on hadoop cluster host cannot load native lib.
If don't use MAPREDUCE-5799 patch, I set my hadoop client's mapred-site.xml with the following
In this way, the YARNRunner will put LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native into
The method ContainerLaunch.expandEnvironment running in NodeManager can translate LD_LIBRARY_PATH={{HADOOP_COMMON_HOME}}/lib/native

The host running NodeManager can find its $HADOOP_COMMON_HOME/lib/native path.

I suggest MRJobConfig.DEFAULT_MAPRED_ADMIN_USER_ENV should be defined as below.
public final String DEFAULT_MAPRED_ADMIN_USER_ENV = 
      Shell.WINDOWS ? 
Please see my patch.

This message was sent by Atlassian JIRA

View raw message