hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ratner, Alan S (IS)" <Alan.Rat...@ngc.com>
Subject HBase Issues (perhaps related to 127.0.0.1)
Date Wed, 21 Nov 2012 20:01:55 GMT
I'd appreciate any suggestions as to how to get HBase up and running.  Right now it dies after a few seconds on all servers.  I am using Hadoop 1.0.4, ZooKeeper 3.4.4 and HBase 0.94.2 on Ubuntu.

History: Yesterday I managed to get HBase 0.94.2 working but only after removing the 127.0.0.1 line from my /etc/hosts file (and synchronizing my clocks).  All was fine until this morning when I realized I could not initiate remote log-ins to my servers (using VNC or NX) until I restored the 127.0.0.1 line in /etc/hosts.  With that restored I am back to a non-working HBase.

With HBase managing ZK I see the following in the HBase Master and ZK logs, respectively:
2012-11-21 13:40:22,236 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase

2012-11-21 13:40:22,122 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running

At roughly the same time (clocks not perfectly synchronized) I see this in a Regionserver log:
2012-11-21 13:40:57,727 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
...
2012-11-21 13:40:57,848 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master

Logs and configuration follows.

Then I tried managing ZK myself and HBase then fails for seemingly different reasons.
2012-11-21 14:46:37,320 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/backup-masters/hadoop1,60000,1353527196915 already deleted, and this is not a retry

2012-11-21 14:46:47,483 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
java.net.ConnectException: Call to hadoop1/127.0.0.1:9000 failed on connection exception: java.net.ConnectException: Connection refused

Both HMaster error logs (self-managed and me-managed ZK) mention the 127.0.0.1 IP address instead of referring to the server by its name (hadoop1) or its true IP address or simply as localhost.

So, start-hbase.sh works OK (HB managing ZK):
ngc@hadoop1:~/hbase-0.94.2$ bin/start-hbase.sh
hadoop1: starting zookeeper, logging to /tmp/hbase-ngc/logs/hbase-ngc-zookeeper-hadoop1.out
hadoop2: starting zookeeper, logging to /tmp/hbase-ngc/logs/hbase-ngc-zookeeper-hadoop2.out
hadoop3: starting zookeeper, logging to /tmp/hbase-ngc/logs/hbase-ngc-zookeeper-hadoop3.out
starting master, logging to /tmp/hbase-ngc/logs/hbase-ngc-master-hadoop1.out
hadoop2: starting regionserver, logging to /tmp/hbase-ngc/logs/hbase-ngc-regionserver-hadoop2.out
hadoop6: starting regionserver, logging to /tmp/hbase-ngc/logs/hbase-ngc-regionserver-hadoop6.out
hadoop3: starting regionserver, logging to /tmp/hbase-ngc/logs/hbase-ngc-regionserver-hadoop3.out
hadoop5: starting regionserver, logging to /tmp/hbase-ngc/logs/hbase-ngc-regionserver-hadoop5.out
hadoop4: starting regionserver, logging to /tmp/hbase-ngc/logs/hbase-ngc-regionserver-hadoop4.out

I have in hbase-site.xml:
  <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
  </property>
      <property>
            <name>hbase.master</name>
            <value>hadoop1:60000</value>
        </property>
  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://hadoop1:9000/hbase</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/tmp/zookeeper_data</value>
  </property>
  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>hadoop1,hadoop2,hadoop3</value>
</property>

I have in hbase-env.sh:
export JAVA_HOME=/home/ngc/jdk1.6.0_25/
export HBASE_CLASSPATH=/home/zookeeper-3.4.4/conf:/home/zookeeper-3.4.4
export HBASE_HEAPSIZE=2000
export HBASE_OPTS="$HBASE_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode"
export HBASE_LOG_DIR=/tmp/hbase-ngc/logs
export HBASE_MANAGES_ZK=true

>From server hadoop1 (running HMaster, ZK, NN, SNN, JT)
Wed Nov 21 13:40:20 EST 2012 Starting master on hadoop1
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 386178
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 386178
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
2012-11-21 13:40:21,410 INFO org.apache.hadoop.hbase.util.VersionInfo: HBase 0.94.2
2012-11-21 13:40:21,410 INFO org.apache.hadoop.hbase.util.VersionInfo: Subversion https://svn.apache.org/repos/asf/hbase/branches/0.94 -r 1395367
2012-11-21 13:40:21,410 INFO org.apache.hadoop.hbase.util.VersionInfo: Compiled by jenkins on Sun Oct  7 19:11:01 UTC 2012
2012-11-21 13:40:21,558 DEBUG org.apache.hadoop.hbase.master.HMaster: Set serverside HConnection retries=100
2012-11-21 13:40:21,823 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 13:40:21,826 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 13:40:21,829 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 13:40:21,833 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 13:40:21,836 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 13:40:21,839 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 13:40:21,842 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 13:40:21,846 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 13:40:21,849 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 13:40:21,852 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 13:40:21,863 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: Initializing RPC Metrics with hostName=HMaster, port=60000
2012-11-21 13:40:22,078 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48 GMT
2012-11-21 13:40:22,078 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=hadoop1
2012-11-21 13:40:22,078 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.6.0_25
2012-11-21 13:40:22,078 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
2012-11-21 13:40:22,078 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/home/ngc/jdk1.6.0_25/jre
2012-11-21 13:40:22,078 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/home/ngc/hbase-0.94.2/conf:/home/ngc/jdk1.6.0_25//lib/tools.jar:/home/ngc/hbase-0.94.2/bin/..:/home/ngc/hbase-0.94.2/bin/../hbase-0.94.2.jar:/home/ngc/hbase-0.94.2/bin/../hbase-0.94.2-tests.jar:/home/ngc/hbase-0.94.2/bin/../lib/activation-1.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/asm-3.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/avro-1.5.3.jar:/home/ngc/hbase-0.94.2/bin/../lib/avro-ipc-1.5.3.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-beanutils-1.7.0.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-cli-1.2.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-codec-1.4.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-collections-3.2.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-configuration-1.6.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-digester-1.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-el-1.0.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-httpclient-3.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-io-2.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-lang-2.5.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-logging-1.1.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-math-2.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-net-1.4.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/core-3.1.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/guava-11.0.2.jar:/home/ngc/hbase-0.94.2/bin/../lib/hadoop-core-1.0.4.jar:/home/ngc/hbase-0.94.2/bin/../lib/high-scale-lib-1.1.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/httpclient-4.1.2.jar:/home/ngc/hbase-0.94.2/bin/../lib/httpcore-4.1.3.jar:/home/ngc/hbase-0.94.2/bin/../lib/jackson-core-asl-1.8.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jackson-jaxrs-1.8.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jackson-mapper-asl-1.8.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jackson-xc-1.8.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jamon-runtime-2.3.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/jasper-compiler-5.5.23.jar:/home/ngc/hbase-0.94.2/bin/../lib/jasper-runtime-5.5.23.jar:/home/ngc/hbase-0.94.2/bin/../lib/jaxb-api-2.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/ngc/hbase-0.94.2/bin/../lib/jersey-core-1.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jersey-json-1.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jersey-server-1.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jettison-1.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/jetty-6.1.26.jar:/home/ngc/hbase-0.94.2/bin/../lib/jetty-util-6.1.26.jar:/home/ngc/hbase-0.94.2/bin/../lib/jruby-complete-1.6.5.jar:/home/ngc/hbase-0.94.2/bin/../lib/jsp-2.1-6.1.14.jar:/home/ngc/hbase-0.94.2/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/ngc/hbase-0.94.2/bin/../lib/jsr305-1.3.9.jar:/home/ngc/hbase-0.94.2/bin/../lib/junit-4.10-HBASE-1.jar:/home/ngc/hbase-0.94.2/bin/../lib/libthrift-0.8.0.jar:/home/ngc/hbase-0.94.2/bin/../lib/log4j-1.2.16.jar:/home/ngc/hbase-0.94.2/bin/../lib/metrics-core-2.1.2.jar:/home/ngc/hbase-0.94.2/bin/../lib/netty-3.2.4.Final.jar:/home/ngc/hbase-0.94.2/bin/../lib/protobuf-java-2.4.0a.jar:/home/ngc/hbase-0.94.2/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/ngc/hbase-0.94.2/bin/../lib/slf4j-api-1.4.3.jar:/home/ngc/hbase-0.94.2/bin/../lib/slf4j-log4j12-1.4.3.jar:/home/ngc/hbase-0.94.2/bin/../lib/snappy-java-1.0.3.2.jar:/home/ngc/hbase-0.94.2/bin/../lib/stax-api-1.0.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/velocity-1.7.jar:/home/ngc/hbase-0.94.2/bin/../lib/xmlenc-0.52.jar:/home/ngc/hbase-0.94.2/bin/../lib/zookeeper-3.4.3.jar:/home/zookeeper-3.4.4/conf:/home/zookeeper-3.4.4:/home/ngc/hadoop-1.0.4/libexec/../conf:/home/ngc/jdk1.6.0_25/lib/tools.jar:/home/ngc/hadoop-1.0.4/libexec/..:/home/ngc/hadoop-1.0.4/libexec/../hadoop-core-1.0.4.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/asm-3.2.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/aspectjrt-1.6.5.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/aspectjtools-1.6.5.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-beanutils-1.7.0.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-beanutils-core-1.8.0.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-cli-1.2.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-codec-1.4.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-collections-3.2.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-configuration-1.6.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-daemon-1.0.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-digester-1.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-el-1.0.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-httpclient-3.0.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-io-2.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-lang-2.4.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-logging-1.1.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-logging-api-1.0.4.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-math-2.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-net-1.4.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/core-3.1.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/hsqldb-1.8.0.10.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jackson-core-asl-1.8.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jasper-compiler-5.5.12.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jasper-runtime-5.5.12.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jdeb-0.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jersey-core-1.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jersey-json-1.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jersey-server-1.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jets3t-0.6.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jetty-6.1.26.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jetty-util-6.1.26.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jsch-0.1.42.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/junit-4.5.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/kfs-0.2.2.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/log4j-1.2.15.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/mockito-all-1.8.5.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/oro-2.0.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/servlet-api-2.5-20081211.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/slf4j-api-1.4.3.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/slf4j-log4j12-1.4.3.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/xmlenc-0.52.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jsp-2.1/jsp-2.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jsp-2.1/jsp-api-2.1.jar
2012-11-21 13:40:22,078 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/home/ngc/hadoop-1.0.4/libexec/../lib/native/Linux-amd64-64:/home/ngc/hbase-0.94.2/bin/../lib/native/Linux-amd64-64
2012-11-21 13:40:22,078 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2012-11-21 13:40:22,078 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2012-11-21 13:40:22,078 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2012-11-21 13:40:22,079 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2012-11-21 13:40:22,079 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.2.0-24-generic
2012-11-21 13:40:22,079 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=ngc
2012-11-21 13:40:22,079 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/ngc
2012-11-21 13:40:22,079 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/ngc/hbase-0.94.2
2012-11-21 13:40:22,080 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop2:2181,hadoop1:2181,hadoop3:2181 sessionTimeout=180000 watcher=master:60000
2012-11-21 13:40:22,097 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /127.0.0.1:2181
2012-11-21 13:40:22,099 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of this process is 742@hadoop1
2012-11-21 13:40:22,106 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:22,106 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:22,110 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1/127.0.0.1:2181, initiating session
2012-11-21 13:40:22,122 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:22,236 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
2012-11-21 13:40:22,236 INFO org.apache.hadoop.hbase.util.RetryCounter: Sleeping 2000ms before retry #1...
2012-11-21 13:40:22,411 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.64.155.53:2181
2012-11-21 13:40:22,411 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:22,411 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:22,412 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181, initiating session
2012-11-21 13:40:22,423 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:22,746 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.64.155.54:2181
2012-11-21 13:40:22,747 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:22,747 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:22,747 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181, initiating session
2012-11-21 13:40:22,748 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:22,967 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.64.155.52:2181
2012-11-21 13:40:22,967 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:22,967 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:22,968 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181, initiating session
2012-11-21 13:40:22,968 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:24,175 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop1/127.0.0.1:2181
2012-11-21 13:40:24,176 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:24,176 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:24,176 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1/127.0.0.1:2181, initiating session
2012-11-21 13:40:24,177 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:24,277 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
2012-11-21 13:40:24,277 INFO org.apache.hadoop.hbase.util.RetryCounter: Sleeping 4000ms before retry #2...
2012-11-21 13:40:24,766 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181
2012-11-21 13:40:24,767 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:24,767 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:24,767 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181, initiating session
2012-11-21 13:40:24,768 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:25,756 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181
2012-11-21 13:40:25,757 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:25,757 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:25,757 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181, initiating session
2012-11-21 13:40:25,757 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:26,597 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181
2012-11-21 13:40:26,597 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:26,597 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:26,598 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181, initiating session
2012-11-21 13:40:26,598 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:27,775 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop1/127.0.0.1:2181
2012-11-21 13:40:27,775 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:27,775 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:27,775 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1/127.0.0.1:2181, initiating session
2012-11-21 13:40:27,776 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:28,317 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181
2012-11-21 13:40:28,318 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:28,318 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:28,318 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181, initiating session
2012-11-21 13:40:28,319 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:28,419 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
2012-11-21 13:40:28,419 INFO org.apache.hadoop.hbase.util.RetryCounter: Sleeping 8000ms before retry #3...
2012-11-21 13:40:29,106 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181
2012-11-21 13:40:29,106 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:29,106 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:29,107 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181, initiating session
2012-11-21 13:40:29,107 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:30,039 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181
2012-11-21 13:40:30,039 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:30,039 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:30,039 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181, initiating session
2012-11-21 13:40:30,040 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:31,283 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop1/127.0.0.1:2181
2012-11-21 13:40:31,283 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:31,283 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:31,283 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1/127.0.0.1:2181, initiating session
2012-11-21 13:40:31,284 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:32,142 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181
2012-11-21 13:40:32,143 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:32,143 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:32,143 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181, initiating session
2012-11-21 13:40:32,144 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:32,479 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181
2012-11-21 13:40:32,480 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:32,480 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:32,480 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181, initiating session
2012-11-21 13:40:32,481 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:33,294 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181
2012-11-21 13:40:33,295 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:33,295 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:33,296 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181, initiating session
2012-11-21 13:40:33,296 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:34,962 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop1/127.0.0.1:2181
2012-11-21 13:40:34,962 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:34,962 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:34,962 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1/127.0.0.1:2181, initiating session
2012-11-21 13:40:34,963 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:35,660 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181
2012-11-21 13:40:35,661 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:35,661 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:35,661 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181, initiating session
2012-11-21 13:40:35,662 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:36,522 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181
2012-11-21 13:40:36,523 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:36,523 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:36,523 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181, initiating session
2012-11-21 13:40:36,524 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:36,625 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
2012-11-21 13:40:36,625 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 3 retries
2012-11-21 13:40:36,626 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
      at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1792)
      at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:146)
      at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
      at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1806)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
      at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
      at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1049)
      at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:193)
      at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:904)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:166)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:159)
      at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:282)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1787)
      ... 5 more


>From server hadoop2 (running regionserver, ZK, DN, TT)
Wed Nov 21 13:40:56 EST 2012 Starting regionserver on hadoop2
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 193105
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 193105
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
2012-11-21 13:40:57,034 INFO org.apache.hadoop.hbase.util.VersionInfo: HBase 0.94.2
2012-11-21 13:40:57,034 INFO org.apache.hadoop.hbase.util.VersionInfo: Subversion https://svn.apache.org/repos/asf/hbase/branches/0.94 -r 1395367
2012-11-21 13:40:57,034 INFO org.apache.hadoop.hbase.util.VersionInfo: Compiled by jenkins on Sun Oct  7 19:11:01 UTC 2012
2012-11-21 13:40:57,172 INFO org.apache.hadoop.hbase.util.ServerCommandLine: vmName=Java HotSpot(TM) 64-Bit Server VM, vmVendor=Sun Microsystems Inc., vmVersion=20.0-b11
2012-11-21 13:40:57,172 INFO org.apache.hadoop.hbase.util.ServerCommandLine: vmInputArguments=[-XX:OnOutOfMemoryError=kill, -9, %p, -Xmx2000m, -XX:+HeapDumpOnOutOfMemoryError, -XX:+UseConcMarkSweepGC, -XX:+CMSIncrementalMode, -XX:+HeapDumpOnOutOfMemoryError, -XX:+UseConcMarkSweepGC, -XX:+CMSIncrementalMode, -Dhbase.log.dir=/tmp/hbase-ngc/logs, -Dhbase.log.file=hbase-ngc-regionserver-hadoop2.log, -Dhbase.home.dir=/home/ngc/hbase-0.94.2/bin/.., -Dhbase.id.str=ngc, -Dhbase.root.logger=INFO,DRFA, -Djava.library.path=/home/ngc/hbase-0.94.2/bin/../lib/native/Linux-amd64-64, -Dhbase.security.logger=INFO,DRFAS]
2012-11-21 13:40:57,222 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Set serverside HConnection retries=100
2012-11-21 13:40:57,469 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-1
2012-11-21 13:40:57,471 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-1
2012-11-21 13:40:57,473 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-1
2012-11-21 13:40:57,475 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-1
2012-11-21 13:40:57,477 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-1
2012-11-21 13:40:57,480 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-1
2012-11-21 13:40:57,482 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-1
2012-11-21 13:40:57,484 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-1
2012-11-21 13:40:57,486 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-1
2012-11-21 13:40:57,488 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-1
2012-11-21 13:40:57,500 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: Initializing RPC Metrics with hostName=HRegionServer, port=60020
2012-11-21 13:40:57,654 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig: Allocating LruBlockCache with maximum size 493.8m
2012-11-21 13:40:57,699 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Installed shutdown hook thread: Shutdownhook:regionserver60020
2012-11-21 13:40:57,701 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48 GMT
2012-11-21 13:40:57,701 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=hadoop2.aj.c2fse.northgrum.com
2012-11-21 13:40:57,701 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.6.0_25
2012-11-21 13:40:57,702 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
2012-11-21 13:40:57,702 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/home/ngc/jdk1.6.0_25/jre
2012-11-21 13:40:57,702 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/home/ngc/hbase-0.94.2/conf:/home/ngc/jdk1.6.0_25//lib/tools.jar:/home/ngc/hbase-0.94.2/bin/..:/home/ngc/hbase-0.94.2/bin/../hbase-0.94.2.jar:/home/ngc/hbase-0.94.2/bin/../hbase-0.94.2-tests.jar:/home/ngc/hbase-0.94.2/bin/../lib/activation-1.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/asm-3.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/avro-1.5.3.jar:/home/ngc/hbase-0.94.2/bin/../lib/avro-ipc-1.5.3.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-beanutils-1.7.0.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-cli-1.2.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-codec-1.4.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-collections-3.2.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-configuration-1.6.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-digester-1.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-el-1.0.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-httpclient-3.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-io-2.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-lang-2.5.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-logging-1.1.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-math-2.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-net-1.4.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/core-3.1.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/guava-11.0.2.jar:/home/ngc/hbase-0.94.2/bin/../lib/hadoop-core-1.0.4.jar:/home/ngc/hbase-0.94.2/bin/../lib/high-scale-lib-1.1.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/httpclient-4.1.2.jar:/home/ngc/hbase-0.94.2/bin/../lib/httpcore-4.1.3.jar:/home/ngc/hbase-0.94.2/bin/../lib/jackson-core-asl-1.8.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jackson-jaxrs-1.8.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jackson-mapper-asl-1.8.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jackson-xc-1.8.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jamon-runtime-2.3.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/jasper-compiler-5.5.23.jar:/home/ngc/hbase-0.94.2/bin/../lib/jasper-runtime-5.5.23.jar:/home/ngc/hbase-0.94.2/bin/../lib/jaxb-api-2.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/ngc/hbase-0.94.2/bin/../lib/jersey-core-1.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jersey-json-1.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jersey-server-1.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jettison-1.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/jetty-6.1.26.jar:/home/ngc/hbase-0.94.2/bin/../lib/jetty-util-6.1.26.jar:/home/ngc/hbase-0.94.2/bin/../lib/jruby-complete-1.6.5.jar:/home/ngc/hbase-0.94.2/bin/../lib/jsp-2.1-6.1.14.jar:/home/ngc/hbase-0.94.2/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/ngc/hbase-0.94.2/bin/../lib/jsr305-1.3.9.jar:/home/ngc/hbase-0.94.2/bin/../lib/junit-4.10-HBASE-1.jar:/home/ngc/hbase-0.94.2/bin/../lib/libthrift-0.8.0.jar:/home/ngc/hbase-0.94.2/bin/../lib/log4j-1.2.16.jar:/home/ngc/hbase-0.94.2/bin/../lib/metrics-core-2.1.2.jar:/home/ngc/hbase-0.94.2/bin/../lib/netty-3.2.4.Final.jar:/home/ngc/hbase-0.94.2/bin/../lib/protobuf-java-2.4.0a.jar:/home/ngc/hbase-0.94.2/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/ngc/hbase-0.94.2/bin/../lib/slf4j-api-1.4.3.jar:/home/ngc/hbase-0.94.2/bin/../lib/slf4j-log4j12-1.4.3.jar:/home/ngc/hbase-0.94.2/bin/../lib/snappy-java-1.0.3.2.jar:/home/ngc/hbase-0.94.2/bin/../lib/stax-api-1.0.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/velocity-1.7.jar:/home/ngc/hbase-0.94.2/bin/../lib/xmlenc-0.52.jar:/home/ngc/hbase-0.94.2/bin/../lib/zookeeper-3.4.3.jar:
2012-11-21 13:40:57,702 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/home/ngc/hbase-0.94.2/bin/../lib/native/Linux-amd64-64
2012-11-21 13:40:57,702 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2012-11-21 13:40:57,702 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2012-11-21 13:40:57,702 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2012-11-21 13:40:57,702 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2012-11-21 13:40:57,702 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.0.0-12-generic
2012-11-21 13:40:57,702 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=ngc
2012-11-21 13:40:57,702 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/ngc
2012-11-21 13:40:57,702 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/ngc/hbase-0.94.2
2012-11-21 13:40:57,703 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop2:2181,hadoop1:2181,hadoop3:2181 sessionTimeout=180000 watcher=regionserver:60020
2012-11-21 13:40:57,718 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.64.155.54:2181
2012-11-21 13:40:57,719 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of this process is 12835@hadoop2
2012-11-21 13:40:57,727 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:57,727 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:57,731 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181, initiating session
2012-11-21 13:40:57,733 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:57,848 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
2012-11-21 13:40:57,849 INFO org.apache.hadoop.hbase.util.RetryCounter: Sleeping 2000ms before retry #1...
2012-11-21 13:40:58,283 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.64.155.53:2181
2012-11-21 13:40:58,283 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:58,283 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:58,283 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181, initiating session
2012-11-21 13:40:58,284 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:58,726 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /127.0.0.1:2181
2012-11-21 13:40:58,726 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:58,726 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:58,726 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1/127.0.0.1:2181, initiating session
2012-11-21 13:40:58,727 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:40:59,367 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.64.155.52:2181
2012-11-21 13:40:59,368 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:40:59,368 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:40:59,368 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181, initiating session
2012-11-21 13:40:59,369 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:00,660 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181
2012-11-21 13:41:00,660 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:00,660 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:00,660 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181, initiating session
2012-11-21 13:41:00,661 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:00,761 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
2012-11-21 13:41:00,762 INFO org.apache.hadoop.hbase.util.RetryCounter: Sleeping 4000ms before retry #2...
2012-11-21 13:41:01,422 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181
2012-11-21 13:41:01,422 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:01,422 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:01,422 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181, initiating session
2012-11-21 13:41:01,423 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:02,369 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop1/127.0.0.1:2181
2012-11-21 13:41:02,370 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:02,370 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:02,370 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1/127.0.0.1:2181, initiating session
2012-11-21 13:41:02,370 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:02,627 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181
2012-11-21 13:41:02,627 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:02,627 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:02,628 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181, initiating session
2012-11-21 13:41:02,628 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:03,968 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181
2012-11-21 13:41:03,968 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:03,969 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:03,969 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181, initiating session
2012-11-21 13:41:03,969 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:04,733 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181
2012-11-21 13:41:04,733 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:04,733 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:04,734 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181, initiating session
2012-11-21 13:41:04,734 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:04,835 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
2012-11-21 13:41:04,835 INFO org.apache.hadoop.hbase.util.RetryCounter: Sleeping 8000ms before retry #3...
2012-11-21 13:41:05,741 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop1/127.0.0.1:2181
2012-11-21 13:41:05,741 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:05,741 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:05,742 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1/127.0.0.1:2181, initiating session
2012-11-21 13:41:05,742 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:06,192 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181
2012-11-21 13:41:06,192 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:06,192 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:06,192 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181, initiating session
2012-11-21 13:41:06,193 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:07,313 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181
2012-11-21 13:41:07,313 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:07,313 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:07,314 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181, initiating session
2012-11-21 13:41:07,314 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:08,272 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181
2012-11-21 13:41:08,273 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:08,273 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:08,273 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181, initiating session
2012-11-21 13:41:08,273 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:09,090 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop1/127.0.0.1:2181
2012-11-21 13:41:09,090 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:09,090 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:09,091 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1/127.0.0.1:2181, initiating session
2012-11-21 13:41:09,091 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:09,710 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181
2012-11-21 13:41:09,711 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:09,711 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:09,711 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181, initiating session
2012-11-21 13:41:09,712 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:11,120 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181
2012-11-21 13:41:11,121 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:11,121 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:11,121 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181, initiating session
2012-11-21 13:41:11,122 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:11,599 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181
2012-11-21 13:41:11,600 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:11,600 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:11,600 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181, initiating session
2012-11-21 13:41:11,600 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:12,320 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop1/127.0.0.1:2181
2012-11-21 13:41:12,320 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:12,320 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:12,321 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1/127.0.0.1:2181, initiating session
2012-11-21 13:41:12,321 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:12,860 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181
2012-11-21 13:41:12,861 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:12,861 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:12,861 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop1.aj.c2fse.northgrum.com/10.64.155.52:2181, initiating session
2012-11-21 13:41:12,862 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:12,962 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
2012-11-21 13:41:12,962 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 3 retries
2012-11-21 13:41:12,963 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: regionserver:60020 Unable to set watcher on znode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
      at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
      at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:166)
      at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:597)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:560)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:669)
      at java.lang.Thread.run(Thread.java:662)
2012-11-21 13:41:12,966 ERROR org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: regionserver:60020 Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
      at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
      at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:166)
      at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:597)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:560)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:669)
      at java.lang.Thread.run(Thread.java:662)
2012-11-21 13:41:12,966 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server hadoop2.aj.c2fse.northgrum.com,60020,1353523257570: Unexpected exception during initialization, aborting
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
      at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1021)
      at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:166)
      at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:230)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:82)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:597)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:560)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:669)
      at java.lang.Thread.run(Thread.java:662)
2012-11-21 13:41:12,969 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
2012-11-21 13:41:12,969 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unexpected exception during initialization, aborting
2012-11-21 13:41:14,834 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181
2012-11-21 13:41:14,834 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:14,834 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:14,834 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181, initiating session
2012-11-21 13:41:14,835 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:15,335 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181
2012-11-21 13:41:15,335 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 13:41:15,335 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 13:41:15,335 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop2.aj.c2fse.northgrum.com/10.64.155.53:2181, initiating session
2012-11-21 13:41:15,336 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2012-11-21 13:41:15,975 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020
2012-11-21 13:41:15,975 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server hadoop2.aj.c2fse.northgrum.com,60020,1353523257570: Initialization of RS failed.  Hence aborting RS.
java.io.IOException: Received the shutdown message while waiting.
      at org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:623)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:598)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:560)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:669)
      at java.lang.Thread.run(Thread.java:662)
2012-11-21 13:41:15,976 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
2012-11-21 13:41:15,976 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Initialization of RS failed.  Hence aborting RS.
2012-11-21 13:41:15,978 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Registered RegionServer MXBean
2012-11-21 13:41:15,980 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-5,5,main]
2012-11-21 13:41:15,980 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook
2012-11-21 13:41:15,981 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown hook thread.
2012-11-21 13:41:15,981 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook finished.

Finally, in the zookeeper log from hadoop1 I have:
Wed Nov 21 13:40:19 EST 2012 Starting zookeeper on hadoop1
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 386178
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 386178
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
2012-11-21 13:40:20,279 INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig: Defaulting to majority quorums
2012-11-21 13:40:20,334 DEBUG org.apache.hadoop.hbase.util.Bytes: preRegister called. Server=com.sun.jmx.mbeanserver.JmxMBeanServer@538f1d7e, name=log4j:logger=org.apache.hadoop.hbase.util.Bytes
2012-11-21 13:40:20,335 DEBUG org.apache.hadoop.hbase.util.VersionInfo: preRegister called. Server=com.sun.jmx.mbeanserver.JmxMBeanServer@538f1d7e, name=log4j:logger=org.apache.hadoop.hbase.util.VersionInfo
2012-11-21 13:40:20,336 DEBUG org.apache.hadoop.hbase.zookeeper.ZKConfig: preRegister called. Server=com.sun.jmx.mbeanserver.JmxMBeanServer@538f1d7e, name=log4j:logger=org.apache.hadoop.hbase.zookeeper.ZKConfig
2012-11-21 13:40:20,336 DEBUG org.apache.hadoop.hbase.HBaseConfiguration: preRegister called. Server=com.sun.jmx.mbeanserver.JmxMBeanServer@538f1d7e, name=log4j:logger=org.apache.hadoop.hbase.HBaseConfiguration
2012-11-21 13:40:20,336 DEBUG org.apache.hadoop.hbase: preRegister called. Server=com.sun.jmx.mbeanserver.JmxMBeanServer@538f1d7e, name=log4j:logger=org.apache.hadoop.hbase
2012-11-21 13:40:20,336 INFO org.apache.zookeeper.server.quorum.QuorumPeerMain: Starting quorum peer
2012-11-21 13:40:20,356 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181
2012-11-21 13:40:20,378 INFO org.apache.zookeeper.server.quorum.QuorumPeer: tickTime set to 3000
2012-11-21 13:40:20,379 INFO org.apache.zookeeper.server.quorum.QuorumPeer: minSessionTimeout set to -1
2012-11-21 13:40:20,379 INFO org.apache.zookeeper.server.quorum.QuorumPeer: maxSessionTimeout set to 180000
2012-11-21 13:40:20,379 INFO org.apache.zookeeper.server.quorum.QuorumPeer: initLimit set to 10
2012-11-21 13:40:20,395 INFO org.apache.zookeeper.server.quorum.QuorumPeer: acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2012-11-21 13:40:20,442 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: My election bind port: 0.0.0.0/0.0.0.0:3888
2012-11-21 13:40:20,456 INFO org.apache.zookeeper.server.quorum.QuorumPeer: LOOKING
2012-11-21 13:40:20,458 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: New election. My id =  0, proposed zxid=0x0
2012-11-21 13:40:20,460 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 0 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 0 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
2012-11-21 13:40:20,464 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Have smaller server identifier, so dropping the connection: (1, 0)
2012-11-21 13:40:20,465 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Have smaller server identifier, so dropping the connection: (2, 0)
2012-11-21 13:40:20,663 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Have smaller server identifier, so dropping the connection: (2, 0)
2012-11-21 13:40:20,663 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Have smaller server identifier, so dropping the connection: (1, 0)
2012-11-21 13:40:20,663 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification time out: 400
2012-11-21 13:40:21,064 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Have smaller server identifier, so dropping the connection: (2, 0)
2012-11-21 13:40:21,065 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Have smaller server identifier, so dropping the connection: (1, 0)
2012-11-21 13:40:21,065 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification time out: 800
2012-11-21 13:40:21,866 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Have smaller server identifier, so dropping the connection: (2, 0)
2012-11-21 13:40:21,866 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Have smaller server identifier, so dropping the connection: (1, 0)
2012-11-21 13:40:21,866 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification time out: 1600
2012-11-21 13:40:22,113 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /127.0.0.1:55216
2012-11-21 13:40:22,122 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2012-11-21 13:40:22,122 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:55216 (no session established for client)
2012-11-21 13:40:22,373 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.64.155.52:60339
2012-11-21 13:40:22,374 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2012-11-21 13:40:22,374 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.64.155.52:60339 (no session established for client)
2012-11-21 13:40:22,968 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.64.155.52:60342
2012-11-21 13:40:22,968 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2012-11-21 13:40:22,968 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.64.155.52:60342 (no session established for client)
2012-11-21 13:40:23,187 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /127.0.0.1:55221
2012-11-21 13:40:23,188 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2012-11-21 13:40:23,188 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:55221 (no session established for client)
2012-11-21 13:40:23,467 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Have smaller server identifier, so dropping the connection: (2, 0)
2012-11-21 13:40:23,467 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Have smaller server identifier, so dropping the connection: (1, 0)
2012-11-21 13:40:23,467 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification time out: 3200
2012-11-21 13:40:24,116 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.64.155.54:35599
2012-11-21 13:40:24,117 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2012-11-21 13:40:24,117 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.64.155.54:35599 (no session established for client)
2012-11-21 13:40:24,176 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /127.0.0.1:55225
...

Here are the logs when I manage ZK myself (showing the 127.0.0.1 problem in /etc/hosts):
Wed Nov 21 14:46:21 EST 2012 Stopping hbase (via master)
Wed Nov 21 14:46:35 EST 2012 Starting master on hadoop1
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 386178
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 386178
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
2012-11-21 14:46:36,405 INFO org.apache.hadoop.hbase.util.VersionInfo: HBase 0.94.2
2012-11-21 14:46:36,405 INFO org.apache.hadoop.hbase.util.VersionInfo: Subversion https://svn.apache.org/repos/asf/hbase/branches/0.94 -r 1395367
2012-11-21 14:46:36,405 INFO org.apache.hadoop.hbase.util.VersionInfo: Compiled by jenkins on Sun Oct  7 19:11:01 UTC 2012
2012-11-21 14:46:36,555 DEBUG org.apache.hadoop.hbase.master.HMaster: Set serverside HConnection retries=100
2012-11-21 14:46:36,822 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 14:46:36,825 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 14:46:36,829 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 14:46:36,832 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 14:46:36,835 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 14:46:36,838 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 14:46:36,842 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 14:46:36,845 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 14:46:36,848 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 14:46:36,851 INFO org.apache.hadoop.ipc.HBaseServer: Starting Thread-2
2012-11-21 14:46:36,862 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: Initializing RPC Metrics with hostName=HMaster, port=60000
2012-11-21 14:46:37,071 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48 GMT
2012-11-21 14:46:37,071 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=hadoop1
2012-11-21 14:46:37,071 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.6.0_25
2012-11-21 14:46:37,071 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
2012-11-21 14:46:37,071 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/home/ngc/jdk1.6.0_25/jre
2012-11-21 14:46:37,071 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/home/ngc/hbase-0.94.2/conf:/home/ngc/jdk1.6.0_25//lib/tools.jar:/home/ngc/hbase-0.94.2/bin/..:/home/ngc/hbase-0.94.2/bin/../hbase-0.94.2.jar:/home/ngc/hbase-0.94.2/bin/../hbase-0.94.2-tests.jar:/home/ngc/hbase-0.94.2/bin/../lib/activation-1.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/asm-3.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/avro-1.5.3.jar:/home/ngc/hbase-0.94.2/bin/../lib/avro-ipc-1.5.3.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-beanutils-1.7.0.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-beanutils-core-1.8.0.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-cli-1.2.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-codec-1.4.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-collections-3.2.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-configuration-1.6.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-digester-1.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-el-1.0.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-httpclient-3.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-io-2.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-lang-2.5.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-logging-1.1.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-math-2.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/commons-net-1.4.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/core-3.1.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/guava-11.0.2.jar:/home/ngc/hbase-0.94.2/bin/../lib/hadoop-core-1.0.4.jar:/home/ngc/hbase-0.94.2/bin/../lib/high-scale-lib-1.1.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/httpclient-4.1.2.jar:/home/ngc/hbase-0.94.2/bin/../lib/httpcore-4.1.3.jar:/home/ngc/hbase-0.94.2/bin/../lib/jackson-core-asl-1.8.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jackson-jaxrs-1.8.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jackson-mapper-asl-1.8.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jackson-xc-1.8.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jamon-runtime-2.3.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/jasper-compiler-5.5.23.jar:/home/ngc/hbase-0.94.2/bin/../lib/jasper-runtime-5.5.23.jar:/home/ngc/hbase-0.94.2/bin/../lib/jaxb-api-2.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/ngc/hbase-0.94.2/bin/../lib/jersey-core-1.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jersey-json-1.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jersey-server-1.8.jar:/home/ngc/hbase-0.94.2/bin/../lib/jettison-1.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/jetty-6.1.26.jar:/home/ngc/hbase-0.94.2/bin/../lib/jetty-util-6.1.26.jar:/home/ngc/hbase-0.94.2/bin/../lib/jruby-complete-1.6.5.jar:/home/ngc/hbase-0.94.2/bin/../lib/jsp-2.1-6.1.14.jar:/home/ngc/hbase-0.94.2/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/ngc/hbase-0.94.2/bin/../lib/jsr305-1.3.9.jar:/home/ngc/hbase-0.94.2/bin/../lib/junit-4.10-HBASE-1.jar:/home/ngc/hbase-0.94.2/bin/../lib/libthrift-0.8.0.jar:/home/ngc/hbase-0.94.2/bin/../lib/log4j-1.2.16.jar:/home/ngc/hbase-0.94.2/bin/../lib/metrics-core-2.1.2.jar:/home/ngc/hbase-0.94.2/bin/../lib/netty-3.2.4.Final.jar:/home/ngc/hbase-0.94.2/bin/../lib/protobuf-java-2.4.0a.jar:/home/ngc/hbase-0.94.2/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/ngc/hbase-0.94.2/bin/../lib/slf4j-api-1.4.3.jar:/home/ngc/hbase-0.94.2/bin/../lib/slf4j-log4j12-1.4.3.jar:/home/ngc/hbase-0.94.2/bin/../lib/snappy-java-1.0.3.2.jar:/home/ngc/hbase-0.94.2/bin/../lib/stax-api-1.0.1.jar:/home/ngc/hbase-0.94.2/bin/../lib/velocity-1.7.jar:/home/ngc/hbase-0.94.2/bin/../lib/xmlenc-0.52.jar:/home/ngc/hbase-0.94.2/bin/../lib/zookeeper-3.4.3.jar:/home/zookeeper-3.4.4/conf:/home/zookeeper-3.4.4:/home/ngc/hadoop-1.0.4/libexec/../conf:/home/ngc/jdk1.6.0_25/lib/tools.jar:/home/ngc/hadoop-1.0.4/libexec/..:/home/ngc/hadoop-1.0.4/libexec/../hadoop-core-1.0.4.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/asm-3.2.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/aspectjrt-1.6.5.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/aspectjtools-1.6.5.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-beanutils-1.7.0.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-beanutils-core-1.8.0.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-cli-1.2.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-codec-1.4.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-collections-3.2.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-configuration-1.6.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-daemon-1.0.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-digester-1.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-el-1.0.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-httpclient-3.0.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-io-2.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-lang-2.4.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-logging-1.1.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-logging-api-1.0.4.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-math-2.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/commons-net-1.4.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/core-3.1.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/hadoop-capacity-scheduler-1.0.4.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/hadoop-fairscheduler-1.0.4.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/hadoop-thriftfs-1.0.4.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/hsqldb-1.8.0.10.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jackson-core-asl-1.8.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jasper-compiler-5.5.12.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jasper-runtime-5.5.12.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jdeb-0.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jersey-core-1.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jersey-json-1.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jersey-server-1.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jets3t-0.6.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jetty-6.1.26.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jetty-util-6.1.26.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jsch-0.1.42.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/junit-4.5.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/kfs-0.2.2.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/log4j-1.2.15.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/mockito-all-1.8.5.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/oro-2.0.8.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/servlet-api-2.5-20081211.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/slf4j-api-1.4.3.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/slf4j-log4j12-1.4.3.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/xmlenc-0.52.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jsp-2.1/jsp-2.1.jar:/home/ngc/hadoop-1.0.4/libexec/../lib/jsp-2.1/jsp-api-2.1.jar
2012-11-21 14:46:37,071 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/home/ngc/hadoop-1.0.4/libexec/../lib/native/Linux-amd64-64:/home/ngc/hbase-0.94.2/bin/../lib/native/Linux-amd64-64
2012-11-21 14:46:37,071 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2012-11-21 14:46:37,071 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2012-11-21 14:46:37,071 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2012-11-21 14:46:37,071 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2012-11-21 14:46:37,071 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.2.0-24-generic
2012-11-21 14:46:37,071 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=ngc
2012-11-21 14:46:37,071 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/home/ngc
2012-11-21 14:46:37,071 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/home/ngc/hbase-0.94.2
2012-11-21 14:46:37,072 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=hadoop2:2181,hadoop1:2181,hadoop3:2181 sessionTimeout=180000 watcher=master:60000
2012-11-21 14:46:37,087 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server /10.64.155.54:2181
2012-11-21 14:46:37,087 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of this process is 12692@hadoop1
2012-11-21 14:46:37,095 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration.
2012-11-21 14:46:37,095 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration.
2012-11-21 14:46:37,098 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181, initiating session
2012-11-21 14:46:37,131 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server hadoop3.aj.c2fse.northgrum.com/10.64.155.54:2181, sessionid = 0x33b247f4c380000, negotiated timeout = 40000
2012-11-21 14:46:37,224 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server Responder: starting
2012-11-21 14:46:37,225 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60000: starting
2012-11-21 14:46:37,240 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60000: starting
2012-11-21 14:46:37,241 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60000: starting
2012-11-21 14:46:37,241 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60000: starting
2012-11-21 14:46:37,241 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60000: starting
2012-11-21 14:46:37,242 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000: starting
2012-11-21 14:46:37,246 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60000: starting
2012-11-21 14:46:37,246 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60000: starting
2012-11-21 14:46:37,247 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000: starting
2012-11-21 14:46:37,247 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60000: starting
2012-11-21 14:46:37,247 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000: starting
2012-11-21 14:46:37,248 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 0 on 60000: starting
2012-11-21 14:46:37,248 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 1 on 60000: starting
2012-11-21 14:46:37,248 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 2 on 60000: starting
2012-11-21 14:46:37,253 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=Master, sessionId=hadoop1,60000,1353527196915
2012-11-21 14:46:37,270 INFO org.apache.hadoop.hbase.metrics: MetricsString added: revision
2012-11-21 14:46:37,270 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUser
2012-11-21 14:46:37,270 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsDate
2012-11-21 14:46:37,270 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsUrl
2012-11-21 14:46:37,270 INFO org.apache.hadoop.hbase.metrics: MetricsString added: date
2012-11-21 14:46:37,270 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsRevision
2012-11-21 14:46:37,270 INFO org.apache.hadoop.hbase.metrics: MetricsString added: user
2012-11-21 14:46:37,270 INFO org.apache.hadoop.hbase.metrics: MetricsString added: hdfsVersion
2012-11-21 14:46:37,270 INFO org.apache.hadoop.hbase.metrics: MetricsString added: url
2012-11-21 14:46:37,270 INFO org.apache.hadoop.hbase.metrics: MetricsString added: version
2012-11-21 14:46:37,270 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2012-11-21 14:46:37,272 INFO org.apache.hadoop.hbase.metrics: new MBeanInfo
2012-11-21 14:46:37,272 INFO org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
2012-11-21 14:46:37,299 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Deleting ZNode for /hbase/backup-masters/hadoop1,60000,1353527196915 from backup master directory
2012-11-21 14:46:37,320 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/backup-masters/hadoop1,60000,1353527196915 already deleted, and this is not a retry
2012-11-21 14:46:37,321 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Master=hadoop1,60000,1353527196915
2012-11-21 14:46:38,475 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop1/127.0.0.1:9000. Already tried 0 time(s).
2012-11-21 14:46:39,476 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop1/127.0.0.1:9000. Already tried 1 time(s).
2012-11-21 14:46:40,477 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop1/127.0.0.1:9000. Already tried 2 time(s).
2012-11-21 14:46:41,477 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop1/127.0.0.1:9000. Already tried 3 time(s).
2012-11-21 14:46:42,478 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop1/127.0.0.1:9000. Already tried 4 time(s).
2012-11-21 14:46:43,478 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop1/127.0.0.1:9000. Already tried 5 time(s).
2012-11-21 14:46:44,479 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop1/127.0.0.1:9000. Already tried 6 time(s).
2012-11-21 14:46:45,479 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop1/127.0.0.1:9000. Already tried 7 time(s).
2012-11-21 14:46:46,480 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop1/127.0.0.1:9000. Already tried 8 time(s).
2012-11-21 14:46:47,480 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop1/127.0.0.1:9000. Already tried 9 time(s).
2012-11-21 14:46:47,483 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
java.net.ConnectException: Call to hadoop1/127.0.0.1:9000 failed on connection exception: java.net.ConnectException: Connection refused
      at org.apache.hadoop.ipc.Client.wrapException(Client.java:1099)
      at org.apache.hadoop.ipc.Client.call(Client.java:1075)
      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
      at $Proxy10.getProtocolVersion(Unknown Source)
      at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
      at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
      at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
      at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
      at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
      at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
      at org.apache.hadoop.hbase.util.FSUtils.getRootDir(FSUtils.java:561)
      at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:94)
      at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:482)
      at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:344)
      at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
      at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
      at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
      at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
      at org.apache.hadoop.ipc.Client.getConnection(Client.java:1206)
      at org.apache.hadoop.ipc.Client.call(Client.java:1050)
      ... 18 more
2012-11-21 14:46:47,485 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2012-11-21 14:46:47,486 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
2012-11-21 14:46:47,486 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60000
2012-11-21 14:46:47,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60000: exiting
2012-11-21 14:46:47,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60000: exiting
2012-11-21 14:46:47,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60000: exiting
2012-11-21 14:46:47,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60000: exiting
2012-11-21 14:46:47,486 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 1 on 60000: exiting
2012-11-21 14:46:47,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60000: exiting
2012-11-21 14:46:47,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60000: exiting
2012-11-21 14:46:47,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000: exiting
2012-11-21 14:46:47,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60000: exiting
2012-11-21 14:46:47,486 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60000
2012-11-21 14:46:47,486 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 0 on 60000: exiting
2012-11-21 14:46:47,486 INFO org.apache.hadoop.ipc.HBaseServer: REPL IPC Server handler 2 on 60000: exiting
2012-11-21 14:46:47,486 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000: exiting
2012-11-21 14:46:47,487 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000: exiting
2012-11-21 14:46:47,488 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2012-11-21 14:46:47,488 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2012-11-21 14:46:47,524 INFO org.apache.zookeeper.ZooKeeper: Session: 0x33b247f4c380000 closed
2012-11-21 14:46:47,524 INFO org.apache.hadoop.hbase.master.HMaster: HMaster main thread exiting
2012-11-21 14:46:47,524 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2012-11-21 14:46:47,524 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: HMaster Aborted
      at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:154)
      at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
      at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1806)

Alan


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message