[ https://issues.apache.org/jira/browse/AMBARI-3074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Wagle updated AMBARI-3074: ------------------------------------ Attachment: AMBARI-3074.patch Ignore failure during NM directory creation, similar in behavior to the DN. > Ambari wont start NodeManager because one of multiple folders not created > ------------------------------------------------------------------------- > > Key: AMBARI-3074 > URL: https://issues.apache.org/jira/browse/AMBARI-3074 > Project: Ambari > Issue Type: Bug > Components: agent > Affects Versions: 1.4.0 > Reporter: Siddharth Wagle > Assignee: Siddharth Wagle > Fix For: 1.4.1 > > Attachments: AMBARI-3074.patch > > > {{yarn-site}} having: > {noformat} > "yarn.nodemanager.local-dirs" : "/grid/0/hadoop/yarn,/grid/1/hadoop/yarn,/grid/2/hadoop/yarn,/grid/3/hadoop/yarn,/grid/4/hadoop/yarn,/grid/5/hadoop/yarn", > "yarn.nodemanager.log-dirs" : "/grid/0/hadoop/yarn,/grid/1/hadoop/yarn,/grid/2/hadoop/yarn,/grid/3/hadoop/yarn,/grid/4/hadoop/yarn,/grid/5/hadoop/yarn", > {noformat} > Now {{/grid/3}} was mounted as read-only due to some disk errors. Though other folders got successfully created, Ambari will not start the NodeManager process. > {noformat} > notice: /Stage[1]/Hdp::Snappy::Package/Hdp::Snappy::Package::Ln[32]/Hdp::Exec[hdp::snappy::package::ln 32]/Exec[hdp::snappy::package::ln 32]/returns: executed successfully > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir -p /grid/3/hadoop/yarn]/Exec[mkdir -p /grid/3/hadoop/yarn]/returns: mkdir: cannot create directory `/grid/3/hadoop': Read-only file system > err: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir -p /grid/3/hadoop/yarn]/Exec[mkdir -p /grid/3/hadoop/yarn]/returns: change from notrun to 0 failed: mkdir -p /grid/3/hadoop/yarn returned 1 instead of one of [0] at /var/lib/ambari-agent/puppet/modules/hdp/manifests/init.pp:479 > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir -p /grid/3/hadoop/yarn]/Anchor[hdp::exec::mkdir -p /grid/3/hadoop/yarn::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Exec[mkdir -p /grid/3/hadoop/yarn]/Anchor[hdp::exec::mkdir -p /grid/3/hadoop/yarn::end]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Directory[/grid/3/hadoop/yarn]/File[/grid/3/hadoop/yarn]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Nodemanager::Create_nm_dirs[/grid/3/hadoop/yarn]/Hdp::Directory_recursive_create[/grid/3/hadoop/yarn]/Hdp::Directory[/grid/3/hadoop/yarn]/File[/grid/3/hadoop/yarn]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[capacity-scheduler]/File[/etc/hadoop/conf/capacity-scheduler.xml]/content: content changed '{md5}e5d17c21c7a5e1db9f3af35cba71df0a' to '{md5}2ca1d267a46f1aecac726caabaa16774' > notice: /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[capacity-scheduler]/File[/etc/hadoop/conf/capacity-scheduler.xml]/owner: owner changed 'hdfs' to 'yarn' > notice: /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[core-site]/File[/etc/hadoop/conf/core-site.xml]/content: content changed '{md5}86d742a780d59a957ea0a283dec03784' to '{md5}8506e4402ba8140ea4f9fed97b6f94e2' > notice: /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[yarn-site]/File[/etc/hadoop/conf/yarn-site.xml]/content: content changed '{md5}d84a967ce47a6b77734ed8f53d817c6e' to '{md5}42940cca6e8f64ae5de50524fb131274' > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::begin]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::begin]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir -p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p /var/log/hadoop-yarn::begin]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir -p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p /var/log/hadoop-yarn::begin]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir -p /var/log/hadoop-yarn]/Exec[mkdir -p /var/log/hadoop-yarn]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir -p /var/log/hadoop-yarn]/Exec[mkdir -p /var/log/hadoop-yarn]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir -p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p /var/log/hadoop-yarn::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Exec[mkdir -p /var/log/hadoop-yarn]/Anchor[hdp::exec::mkdir -p /var/log/hadoop-yarn::end]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Directory[/var/log/hadoop-yarn]/File[/var/log/hadoop-yarn]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/log/hadoop-yarn]/Hdp::Directory[/var/log/hadoop-yarn]/File[/var/log/hadoop-yarn]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir -p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p /var/run/hadoop-yarn/yarn::begin]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir -p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p /var/run/hadoop-yarn/yarn::begin]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir -p /var/run/hadoop-yarn/yarn]/Exec[mkdir -p /var/run/hadoop-yarn/yarn]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir -p /var/run/hadoop-yarn/yarn]/Exec[mkdir -p /var/run/hadoop-yarn/yarn]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir -p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p /var/run/hadoop-yarn/yarn::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Exec[mkdir -p /var/run/hadoop-yarn/yarn]/Anchor[hdp::exec::mkdir -p /var/run/hadoop-yarn/yarn::end]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Directory[/var/run/hadoop-yarn/yarn]/File[/var/run/hadoop-yarn/yarn]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Directory_recursive_create[/var/run/hadoop-yarn/yarn]/Hdp::Directory[/var/run/hadoop-yarn/yarn]/File[/var/run/hadoop-yarn/yarn]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su - yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']/Anchor[hdp::exec::su - yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager'::begin]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su - yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']/Anchor[hdp::exec::su - yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager'::begin]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su - yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']/Exec[su - yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su - yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']/Exec[su - yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su - yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']/Anchor[hdp::exec::su - yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager'::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[su - yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager']/Anchor[hdp::exec::su - yarn -c 'export HADOOP_LIBEXEC_DIR=/usr/lib/hadoop/libexec && /usr/lib/hadoop-yarn/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager'::end]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]/Anchor[hdp::exec::sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1::begin]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]/Anchor[hdp::exec::sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1::begin]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]/Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]/Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]/Anchor[hdp::exec::sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Hdp::Exec[sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1]/Anchor[hdp::exec::sleep 5; ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1::end]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Hdp-yarn::Service[nodemanager]/Anchor[hdp-yarn::service::nodemanager::end]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Nodemanager/Anchor[hdp-yarn::nodemanager::end]: Dependency Exec[mkdir -p /grid/3/hadoop/yarn] has failures: true > warning: /Stage[2]/Hdp-yarn::Nodemanager/Anchor[hdp-yarn::nodemanager::end]: Skipping because of failed dependencies > notice: /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[mapred-site]/File[/etc/hadoop/conf/mapred-site.xml]/content: content changed '{md5}093cb1899b3c3b9dc4a7c1c93729c18b' to '{md5}4c462999cc47e6f6ba0e6381d71d81ba' > notice: /Stage[2]/Hdp-yarn::Initialize/Hdp-yarn::Generate_common_configs[yarn-common-configs]/Configgenerator::Configfile[mapred-site]/File[/etc/hadoop/conf/mapred-site.xml]/owner: owner changed 'mapred' to 'yarn' > notice: Finished catalog run in 2.39 seconds > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira