hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elliott Clark <ecl...@apache.org>
Subject Re: Trouble using copyTable to seed existing table to replication peer
Date Sat, 27 Sep 2014 01:14:22 GMT
Do any map-reduce jobs work on this cluster ?

On Fri, Sep 26, 2014 at 5:13 PM, iain wright <iainwrig@gmail.com> wrote:

> Forgot to include the actual copy table command:
> [hbase@master2 bin]$ ./hbase org.apache.hadoop.hbase.mapreduce.CopyTable
> --starttime=1409128964 --peer.adr=master0.hbasex1.test.cloud.domain.tv,
> master1.hbasex1.test.cloud.v.tv,master2.hbasex1.test.cloud.domain.tv:2181
> :/hbase
> moderation
>
> Replication is working well btw for new data.
>
> Thanks,
>
>
> --
> Iain Wright
>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>
> On Fri, Sep 26, 2014 at 5:11 PM, iain wright <iainwrig@gmail.com> wrote:
>
> > Hi folks,
> >
> > I'm having trouble using copyTable to seed an existing tables data to a
> > replication peer. Surely its an oversight in configuration on our part,
> but
> > I've scoured the web and doc's for a couple days now.
> >
> > We have been able to run these jobs with success (perhaps they don't
> > require localization of any jars since the jars are already in $PATH?):
> > ./hadoop org.apache.hadoop.fs.TestDFSIO -write -nrFiles 5 -fileSize 5GB
> > ./hbase org.apache.hadoop.hbase.PerformanceEvaluation  randomWrite 5
> >
> > When running copyTable it seems like the resources are placed on the yarn
> > application manager but not localized to the nodemanagers.
> >
> >  I've tried several settings for yarn.app.mapreduce.am.staging-dir and in
> > the below iteration im trying the defaults, the path exists with correct
> > permissions on each NM/AM
> >
> > hadoop.tmp.dir is specified in core-site.xml
> >
> > Here is the full command we are running:
> >
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> >
> [jar:file:/usr/local/hbase-0.98.6.1-1a/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> >
> [jar:file:/usr/local/hadoop-2.5.0-1a/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> > explanation.
> > 2014-09-26 23:51:24,141 WARN  [main] util.NativeCodeLoader: Unable to
> load
> > native-hadoop library for your platform... using builtin-java classes
> where
> > applicable
> > 2014-09-26 23:51:24,863 WARN  [main] shortcircuit.DomainSocketFactory:
> The
> > short-circuit local reads feature cannot be used because libhadoop cannot
> > be loaded.
> > 2014-09-26 23:51:24,910 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.hbase.HConstants, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar
> > 2014-09-26 23:51:24,911 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/hbase-protocol-0.98.6.1-hadoop2.jar
> > 2014-09-26 23:51:24,912 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.hbase.client.Put, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/hbase-client-0.98.6.1-hadoop2.jar
> > 2014-09-26 23:51:24,912 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.hbase.CompatibilityFactory, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar
> > 2014-09-26 23:51:24,913 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar
> > 2014-09-26 23:51:24,914 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.zookeeper.ZooKeeper, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/zookeeper-3.4.6.jar
> > 2014-09-26 23:51:24,914 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.jboss.netty.channel.ChannelFactory, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/netty-3.6.6.Final.jar
> > 2014-09-26 23:51:24,915 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class com.google.protobuf.Message, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/protobuf-java-2.5.0.jar
> > 2014-09-26 23:51:24,916 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class com.google.common.collect.Lists, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/guava-12.0.1.jar
> > 2014-09-26 23:51:24,916 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.cloudera.htrace.Trace, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/htrace-core-2.04.jar
> > 2014-09-26 23:51:24,917 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.cliffc.high_scale_lib.Counter, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/high-scale-lib-1.1.1.jar
> > 2014-09-26 23:51:24,921 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.io.LongWritable, using jar
> > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar
> > 2014-09-26 23:51:24,922 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.io.Text, using jar
> > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar
> > 2014-09-26 23:51:24,923 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.hbase.mapreduce.TableInputFormat, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar
> > 2014-09-26 23:51:24,923 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.io.LongWritable, using jar
> > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar
> > 2014-09-26 23:51:24,924 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.io.Text, using jar
> > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar
> > 2014-09-26 23:51:24,925 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.mapreduce.lib.output.TextOutputFormat, using jar
> >
> /usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar
> > 2014-09-26 23:51:24,926 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using
> jar
> >
> /usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar
> > 2014-09-26 23:51:24,952 INFO  [main] Configuration.deprecation:
> > io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
> > 2014-09-26 23:51:24,963 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.hbase.HConstants, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar
> > 2014-09-26 23:51:24,963 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.hbase.protobuf.generated.ClientProtos, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/hbase-protocol-0.98.6.1-hadoop2.jar
> > 2014-09-26 23:51:24,964 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.hbase.client.Put, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/hbase-client-0.98.6.1-hadoop2.jar
> > 2014-09-26 23:51:24,964 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.hbase.CompatibilityFactory, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar
> > 2014-09-26 23:51:24,965 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.hbase.mapreduce.TableMapper, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar
> > 2014-09-26 23:51:24,966 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.zookeeper.ZooKeeper, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/zookeeper-3.4.6.jar
> > 2014-09-26 23:51:24,966 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.jboss.netty.channel.ChannelFactory, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/netty-3.6.6.Final.jar
> > 2014-09-26 23:51:24,967 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class com.google.protobuf.Message, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/protobuf-java-2.5.0.jar
> > 2014-09-26 23:51:24,967 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class com.google.common.collect.Lists, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/guava-12.0.1.jar
> > 2014-09-26 23:51:24,968 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.cloudera.htrace.Trace, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/htrace-core-2.04.jar
> > 2014-09-26 23:51:24,968 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.cliffc.high_scale_lib.Counter, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/high-scale-lib-1.1.1.jar
> > 2014-09-26 23:51:24,971 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar
> > 2014-09-26 23:51:24,972 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.io.Writable, using jar
> > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar
> > 2014-09-26 23:51:24,972 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.hbase.mapreduce.TableInputFormat, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar
> > 2014-09-26 23:51:24,973 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.hbase.io.ImmutableBytesWritable, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/hbase-common-0.98.6.1-hadoop2.jar
> > 2014-09-26 23:51:24,973 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.io.Writable, using jar
> > /usr/local/hadoop-2.5.0-1a/share/hadoop/common/hadoop-common-2.5.0.jar
> > 2014-09-26 23:51:24,974 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.hbase.mapreduce.TableOutputFormat, using jar
> > /usr/local/hbase-0.98.6.1-1a/lib/hbase-server-0.98.6.1-hadoop2.jar
> > 2014-09-26 23:51:24,975 DEBUG [main] mapreduce.TableMapReduceUtil: For
> > class org.apache.hadoop.mapreduce.lib.partition.HashPartitioner, using
> jar
> >
> /usr/local/hadoop-2.5.0-1a/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar
> > 2014-09-26 23:51:25,035 INFO  [main] client.RMProxy: Connecting to
> > ResourceManager at
> master2.hbase1a.test.cloud.domain.tv/172.16.0.145:8032
> > 2014-09-26 23:51:25,191 INFO  [main] Configuration.deprecation:
> > io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
> > 2014-09-26 23:51:25,247 INFO  [main] zookeeper.RecoverableZooKeeper:
> > Process identifier=hconnection-0x6bf3ede0 connecting to ZooKeeper
> ensemble=
> > master1.hbasex1.test.cloud.domain.tv:2181,
> > master0.hbasex1.test.cloud.domain.tv:2181,
> > master2.hbasex1.test.cloud.domain.tv:2181
> > 2014-09-26 23:51:25,252 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> GMT
> > 2014-09-26 23:51:25,252 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:host.name=master2.hbase1a.test.cloud.domain.tv
> > 2014-09-26 23:51:25,252 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:java.version=1.7.0_65
> > 2014-09-26 23:51:25,252 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:java.vendor=Oracle Corporation
> > 2014-09-26 23:51:25,252 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre
> > 2014-09-26 23:51:25,252 INFO  [main] zookeeper.ZooKeeper: Client
> >
> environment:java.class.path=/usr/local/hbase/bin/../conf:/usr/lib/jvm/java/lib/tools.jar:/usr/local/hbase/bin/..:/usr/local/hbase/bin/../lib/activation-1.1.jar:/usr/local/hbase/bin/../lib/aopalliance-1.0.jar:/usr/local/hbase/bin/../lib/asm-3.1.jar:/usr/local/hbase/bin/../lib/avro-1.7.4.jar:/usr/local/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/usr/local/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase/bin/../lib/commons-cli-1.2.jar:/usr/local/hbase/bin/../lib/commons-codec-1.7.jar:/usr/local/hbase/bin/../lib/commons-collections-3.2.1.jar:/usr/local/hbase/bin/../lib/commons-compress-1.4.1.jar:/usr/local/hbase/bin/../lib/commons-configuration-1.6.jar:/usr/local/hbase/bin/../lib/commons-daemon-1.0.13.jar:/usr/local/hbase/bin/../lib/commons-digester-1.8.jar:/usr/local/hbase/bin/../lib/commons-el-1.0.jar:/usr/local/hbase/bin/../lib/commons-httpclient-3.1.jar:/usr/local/hbase/bin/../lib/commons-io-2.4.jar:/usr/local/hbase/bin/../lib/commons-lang-2.6.jar:/usr/local/hbase/bin/../lib/commons-logging-1.1.1.jar:/usr/local/hbase/bin/../lib/commons-math-2.1.jar:/usr/local/hbase/bin/../lib/commons-net-3.1.jar:/usr/local/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr/local/hbase/bin/../lib/gmbal-api-only-3.0.0-b023.jar:/usr/local/hbase/bin/../lib/grizzly-framework-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-server-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-http-servlet-2.1.2.jar:/usr/local/hbase/bin/../lib/grizzly-rcm-2.1.2.jar:/usr/local/hbase/bin/../lib/guava-12.0.1.jar:/usr/local/hbase/bin/../lib/guice-3.0.jar:/usr/local/hbase/bin/../lib/guice-servlet-3.0.jar:/usr/local/hbase/bin/../lib/hamcrest-core-1.3.jar:/usr/local/hbase/bin/../lib/hbase-client-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-common-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-common-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-examples-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-hadoop2-compat-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-hadoop-compat-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-it-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-it-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-prefix-tree-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-protocol-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-server-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-server-0.98.6.1-hadoop2-tests.jar:/usr/local/hbase/bin/../lib/hbase-shell-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-testing-util-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/hbase-thrift-0.98.6.1-hadoop2.jar:/usr/local/hbase/bin/../lib/high-scale-lib-1.1.1.jar:/usr/local/hbase/bin/../lib/htrace-core-2.04.jar:/usr/local/hbase/bin/../lib/httpclient-4.1.3.jar:/usr/local/hbase/bin/../lib/httpcore-4.1.3.jar:/usr/local/hbase/bin/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase/bin/../lib/jackson-xc-1.8.8.jar:/usr/local/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/usr/local/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/local/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/usr/local/hbase/bin/../lib/javax.inject-1.jar:/usr/local/hbase/bin/../lib/javax.servlet-3.1.jar:/usr/local/hbase/bin/../lib/javax.servlet-api-3.0.1.jar:/usr/local/hbase/bin/../lib/jaxb-api-2.2.2.jar:/usr/local/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase/bin/../lib/jersey-client-1.9.jar:/usr/local/hbase/bin/../lib/jersey-core-1.8.jar:/usr/local/hbase/bin/../lib/jersey-grizzly2-1.9.jar:/usr/local/hbase/bin/../lib/jersey-guice-1.9.jar:/usr/local/hbase/bin/../lib/jersey-json-1.8.jar:/usr/local/hbase/bin/../lib/jersey-server-1.8.jar:/usr/local/hbase/bin/../lib/jersey-test-framework-core-1.9.jar:/usr/local/hbase/bin/../lib/jersey-test-framework-grizzly2-1.9.jar:/usr/local/hbase/bin/../lib/jets3t-0.6.1.jar:/usr/local/hbase/bin/../lib/jettison-1.3.1.jar:/usr/local/hbase/bin/../lib/jetty-6.1.26.jar:/usr/local/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/usr/local/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/local/hbase/bin/../lib/jruby-complete-1.6.8.jar:/usr/local/hbase/bin/../lib/jsch-0.1.42.jar:/usr/local/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase/bin/../lib/jsr305-1.3.9.jar:/usr/local/hbase/bin/../lib/junit-4.11.jar:/usr/local/hbase/bin/../lib/libthrift-0.9.0.jar:/usr/local/hbase/bin/../lib/log4j-1.2.17.jar:/usr/local/hbase/bin/../lib/management-api-3.0.0-b012.jar:/usr/local/hbase/bin/../lib/metrics-core-2.2.0.jar:/usr/local/hbase/bin/../lib/netty-3.6.6.Final.jar:/usr/local/hbase/bin/../lib/paranamer-2.3.jar:/usr/local/hbase/bin/../lib/protobuf-java-2.5.0.jar:/usr/local/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase/bin/../lib/slf4j-api-1.6.4.jar:/usr/local/hbase/bin/../lib/slf4j-log4j12-1.6.4.jar:/usr/local/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/usr/local/hbase/bin/../lib/xmlenc-0.52.jar:/usr/local/hbase/bin/../lib/xz-1.0.jar:/usr/local/hbase/bin/../lib/zookeeper-3.4.6.jar:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-annotations-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/common/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/hadoop/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/hadoop-nfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jline-0.9.94.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/local/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0-tests.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar:/usr/local/hadoop/contrib/capacity-scheduler/*.jar:/usr/local/hadoop/etc/hadoop/
> > 2014-09-26 23:51:25,253 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:java.library.path=/usr/local/hadoop/lib
> > 2014-09-26 23:51:25,253 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:java.io.tmpdir=/tmp
> > 2014-09-26 23:51:25,253 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:java.compiler=<NA>
> > 2014-09-26 23:51:25,253 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:os.name=Linux
> > 2014-09-26 23:51:25,253 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:os.arch=amd64
> > 2014-09-26 23:51:25,253 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:os.version=3.10.42-52.145.amzn1.x86_64
> > 2014-09-26 23:51:25,253 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:user.name=hbase
> > 2014-09-26 23:51:25,253 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:user.home=/home/hbase
> > 2014-09-26 23:51:25,253 INFO  [main] zookeeper.ZooKeeper: Client
> > environment:user.dir=/usr/local/hbase-0.98.6.1-1a/bin
> > 2014-09-26 23:51:25,254 INFO  [main] zookeeper.ZooKeeper: Initiating
> > client connection, connectString=
> master1.hbasex1.test.cloud.domain.tv:2181
> > ,master0.hbasex1.test.cloud.domain.tv:2181,
> > master2.hbasex1.test.cloud.domain.tv:2181 sessionTimeout=90000
> > watcher=hconnection-0x6bf3ede0, quorum=
> > master1.hbasex1.test.cloud.domain.tv:2181,
> > master0.hbasex1.test.cloud.domain.tv:2181,
> > master2.hbasex1.test.cloud.domain.tv:2181, baseZNode=/hbase
> > 2014-09-26 23:51:25,270 INFO  [main-SendThread(
> > master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn:
> Opening
> > socket connection to server
> > master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181. Will not attempt
> > to authenticate using SASL (unknown error)
> > 2014-09-26 23:51:25,276 INFO  [main-SendThread(
> > master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Socket
> > connection established to
> > master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181, initiating
> session
> > 2014-09-26 23:51:25,282 INFO  [main-SendThread(
> > master1.hbasex1.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn:
> Session
> > establishment complete on server
> > master1.hbasex1.test.cloud.domain.tv/172.16.1.16:2181, sessionid =
> > 0x248afb6109b007f, negotiated timeout = 40000
> > 2014-09-26 23:51:25,308 INFO  [main] mapreduce.TableOutputFormat: Created
> > table instance for moderation
> > 2014-09-26 23:51:25,469 INFO  [main] zookeeper.RecoverableZooKeeper:
> > Process identifier=hconnection-0xa7a371 connecting to ZooKeeper ensemble=
> > master1.hbase1a.test.cloud.domain.tv:2181,
> > master0.hbase1a.test.cloud.domain.tv:2181,
> > master2.hbase1a.test.cloud.domain.tv:2181
> > 2014-09-26 23:51:25,469 INFO  [main] zookeeper.ZooKeeper: Initiating
> > client connection, connectString=
> master1.hbase1a.test.cloud.domain.tv:2181
> > ,master0.hbase1a.test.cloud.domain.tv:2181,
> > master2.hbase1a.test.cloud.domain.tv:2181 sessionTimeout=90000
> > watcher=hconnection-0xa7a371, quorum=
> > master1.hbase1a.test.cloud.domain.tv:2181,
> > master0.hbase1a.test.cloud.domain.tv:2181,
> > master2.hbase1a.test.cloud.domain.tv:2181, baseZNode=/hbase
> > 2014-09-26 23:51:25,470 INFO  [main-SendThread(
> > master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn:
> Opening
> > socket connection to server
> > master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181. Will not attempt
> > to authenticate using SASL (unknown error)
> > 2014-09-26 23:51:25,470 INFO  [main-SendThread(
> > master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn: Socket
> > connection established to
> > master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181, initiating
> session
> > 2014-09-26 23:51:25,472 INFO  [main-SendThread(
> > master2.hbase1a.test.cloud.domain.tv:2181)] zookeeper.ClientCnxn:
> Session
> > establishment complete on server
> > master2.hbase1a.test.cloud.domain.tv/172.16.0.145:2181, sessionid =
> > 0x34877671747028e, negotiated timeout = 40000
> > 2014-09-26 23:51:25,490 INFO  [main] util.RegionSizeCalculator:
> > Calculating region sizes for table "moderation".
> > 2014-09-26 23:51:25,686 DEBUG [main] util.RegionSizeCalculator: Region
> > moderation,,1411774704618.68a1839eacfa355028522286caaf7cd5. has size 0
> > 2014-09-26 23:51:25,686 DEBUG [main] util.RegionSizeCalculator: Region
> > sizes calculated
> > 2014-09-26 23:51:25,709 DEBUG [main] client.ClientSmallScanner: Finished
> > with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1',
> > STARTKEY => '', ENDKEY => ''}
> > 2014-09-26 23:51:25,722 DEBUG [main] mapreduce.TableInputFormatBase:
> > getSplits: split -> 0 -> ip-172-16-0-247.us-west-1.compute.internal:,
> > 2014-09-26 23:51:25,739 INFO  [main] mapreduce.JobSubmitter: number of
> > splits:1
> > 2014-09-26 23:51:25,748 INFO  [main] Configuration.deprecation:
> > io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
> > 2014-09-26 23:51:25,895 INFO  [main] mapreduce.JobSubmitter: Submitting
> > tokens for job: job_1411771844660_0004
> > 2014-09-26 23:51:26,033 INFO  [main] impl.YarnClientImpl: Submitted
> > application application_1411771844660_0004
> > 2014-09-26 23:51:26,066 INFO  [main] mapreduce.Job: The url to track the
> > job:
> >
> http://master2.hbase1a.test.cloud.domain.tv:8088/proxy/application_1411771844660_0004/
> > 2014-09-26 23:51:26,067 INFO  [main] mapreduce.Job: Running job:
> > job_1411771844660_0004
> > 2014-09-26 23:51:29,082 INFO  [main] mapreduce.Job: Job
> > job_1411771844660_0004 running in uber mode : false
> > 2014-09-26 23:51:29,084 INFO  [main] mapreduce.Job:  map 0% reduce 0%
> > 2014-09-26 23:51:29,094 INFO  [main] mapreduce.Job: Job
> > job_1411771844660_0004 failed with state FAILED due to: Application
> > application_1411771844660_0004 failed 2 times due to AM Container for
> > appattempt_1411771844660_0004_000002 exited with  exitCode: -1000 due to:
> > File
> >
> file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar
> > does not exist
> > .Failing this attempt.. Failing the application.
> > 2014-09-26 23:51:29,111 INFO  [main] mapreduce.Job: Counters: 0
> >
> > The directory on the AM (master) from which the command is initiated as
> > hbase user:
> > [hbase@master2 bin]$ ls -lah
> > /tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/
> > total 3.7M
> > drwx------  3 hbase hbase  4.0K Sep 26 23:51 .
> > drwx------ 10 hbase hadoop 4.0K Sep 26 23:51 ..
> > -rw-r--r--  1 hbase hbase  3.5M Sep 26 23:51 job.jar
> > -rw-rw-r--  1 hbase hbase   28K Sep 26 23:51 .job.jar.crc
> > -rw-r--r--  1 hbase hbase   111 Sep 26 23:51 job.split
> > -rw-rw-r--  1 hbase hbase    12 Sep 26 23:51 .job.split.crc
> > -rw-r--r--  1 hbase hbase    56 Sep 26 23:51 job.splitmetainfo
> > -rw-rw-r--  1 hbase hbase    12 Sep 26 23:51 .job.splitmetainfo.crc
> > -rw-r--r--  1 hbase hbase  102K Sep 26 23:51 job.xml
> > -rw-rw-r--  1 hbase hbase   824 Sep 26 23:51 .job.xml.crc
> > drwx------  2 hbase hbase  4.0K Sep 26 23:51 libjars
> >
> > The same directory is empty on each NM.
> >
> > Heres also the log from a node that got the job/application:
> > 2014-09-26 23:51:26,635 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
> > Auth successful for appattempt_1411771844660_0004_000001 (auth:SIMPLE)
> > 2014-09-26 23:51:26,639 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> > Start request for container_1411771844660_0004_01_000001 by user hbase
> > 2014-09-26 23:51:26,639 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> > Creating a new application reference for app
> application_1411771844660_0004
> > 2014-09-26 23:51:26,639 INFO
> > org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hbase
> >  IP=172.16.0.145 OPERATION=Start Container Request
> > TARGET=ContainerManageImpl      RESULT=SUCCESS
> >  APPID=application_1411771844660_0004
> >  CONTAINERID=container_1411771844660_0004_01_000001
> > 2014-09-26 23:51:26,639 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> > Application application_1411771844660_0004 transitioned from NEW to
> INITING
> > 2014-09-26 23:51:26,639 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> > Adding container_1411771844660_0004_01_000001 to application
> > application_1411771844660_0004
> > 2014-09-26 23:51:26,639 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> > Application application_1411771844660_0004 transitioned from INITING to
> > RUNNING
> > 2014-09-26 23:51:26,640 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> > Container container_1411771844660_0004_01_000001 transitioned from NEW to
> > LOCALIZING
> > 2014-09-26 23:51:26,640 INFO
> > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
> Got
> > event CONTAINER_INIT for appId application_1411771844660_0004
> > 2014-09-26 23:51:26,640 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> > Resource
> >
> file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar
> > transitioned from INIT to DOWNLOADING
> > 2014-09-26 23:51:26,640 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> > Resource
> >
> file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.splitmetainfo
> > transitioned from INIT to DOWNLOADING
> > 2014-09-26 23:51:26,640 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> > Resource
> >
> file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.split
> > transitioned from INIT to DOWNLOADING
> > 2014-09-26 23:51:26,641 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> > Resource
> >
> file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.xml
> > transitioned from INIT to DOWNLOADING
> > 2014-09-26 23:51:26,641 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> > Created localizer for container_1411771844660_0004_01_000001
> > 2014-09-26 23:51:26,645 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> > Writing credentials to the nmPrivate file
> >
> /data0/hadoop/yarn/local/nmPrivate/container_1411771844660_0004_01_000001.tokens.
> > Credentials list:
> > 2014-09-26 23:51:26,648 INFO
> > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> > Initializing user hbase
> > 2014-09-26 23:51:26,661 INFO
> > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> Copying
> > from
> >
> /data0/hadoop/yarn/local/nmPrivate/container_1411771844660_0004_01_000001.tokens
> > to
> >
> /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001.tokens
> > 2014-09-26 23:51:26,661 INFO
> > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: CWD
> set
> > to
> >
> /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004
> > =
> >
> file:/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004
> > 2014-09-26 23:51:26,696 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> > DEBUG: FAILED {
> >
> file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar,
> > 1411775485000, PATTERN, (?:classes/|lib/).* }, File
> >
> file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar
> > does not exist
> > 2014-09-26 23:51:26,697 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource:
> > Resource
> >
> file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar(->/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/filecache/10/job.jar)
> > transitioned from DOWNLOADING to FAILED
> > 2014-09-26 23:51:26,697 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> > Container container_1411771844660_0004_01_000001 transitioned from
> > LOCALIZING to LOCALIZATION_FAILED
> > 2014-09-26 23:51:26,697 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourcesTrackerImpl:
> > Container container_1411771844660_0004_01_000001 sent RELEASE event on a
> > resource request {
> >
> file:/tmp/hadoop-yarn/staging/hbase/.staging/job_1411771844660_0004/job.jar,
> > 1411775485000, PATTERN, (?:classes/|lib/).* } not present in cache.
> > 2014-09-26 23:51:26,698 INFO
> > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> > Deleting absolute path :
> >
> /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001
> > 2014-09-26 23:51:26,698 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
> > Unknown localizer with localizerId container_1411771844660_0004_01_000001
> > is sending heartbeat. Ordering it to DIE
> > 2014-09-26 23:51:26,698 WARN
> > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> delete
> > returned false for path:
> >
> [/data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001]
> > 2014-09-26 23:51:26,698 INFO
> > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> > Deleting absolute path :
> >
> /data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001
> > 2014-09-26 23:51:26,699 WARN
> > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> delete
> > returned false for path:
> >
> [/data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004/container_1411771844660_0004_01_000001]
> > 2014-09-26 23:51:26,699 WARN
> > org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hbase
> >  OPERATION=Container Finished - Failed   TARGET=ContainerImpl
> >  RESULT=FAILURE  DESCRIPTION=Container failed with state:
> > LOCALIZATION_FAILED    APPID=application_1411771844660_0004
> >  CONTAINERID=container_1411771844660_0004_01_000001
> > 2014-09-26 23:51:26,699 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
> > Container container_1411771844660_0004_01_000001 transitioned from
> > LOCALIZATION_FAILED to DONE
> > 2014-09-26 23:51:26,699 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> > Removing container_1411771844660_0004_01_000001 from application
> > application_1411771844660_0004
> > 2014-09-26 23:51:26,699 INFO
> > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
> Got
> > event CONTAINER_STOP for appId application_1411771844660_0004
> > 2014-09-26 23:51:27,629 INFO
> > org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed
> > completed containers from NM context:
> > [container_1411771844660_0004_01_000001]
> > 2014-09-26 23:51:28,252 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
> > Stopping resource-monitoring for container_1411771844660_0004_01_000001
> > 2014-09-26 23:51:29,631 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> > Application application_1411771844660_0004 transitioned from RUNNING to
> > APPLICATION_RESOURCES_CLEANINGUP
> > 2014-09-26 23:51:29,632 INFO
> > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> > Deleting absolute path :
> >
> /data0/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004
> > 2014-09-26 23:51:29,632 INFO
> > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
> > Deleting absolute path :
> >
> /data1/hadoop/yarn/local/usercache/hbase/appcache/application_1411771844660_0004
> > 2014-09-26 23:51:29,632 INFO
> > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices:
> Got
> > event APPLICATION_STOP for appId application_1411771844660_0004
> > 2014-09-26 23:51:29,632 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
> > Application application_1411771844660_0004 transitioned from
> > APPLICATION_RESOURCES_CLEANINGUP to FINISHED
> > 2014-09-26 23:51:29,632 INFO
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler:
> > Scheduling Log Deletion for application: application_1411771844660_0004,
> > with delay of 10800 seconds
> >
> > Thank you in advance for any advice
> >
> > --
> > Iain Wright
> >
> > This email message is confidential, intended only for the recipient(s)
> > named above and may contain information that is privileged, exempt from
> > disclosure under applicable law. If you are not the intended recipient,
> do
> > not disclose or disseminate the message to anyone except the intended
> > recipient. If you have received this message in error, or are not the
> named
> > recipient(s), please immediately notify the sender by return email, and
> > delete all copies of this message.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message