hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mathias De Maré <mathias.dem...@gmail.com>
Subject HBase cluster -- 2 machines -- ZooKeeper problems when launching jobs
Date Fri, 21 Aug 2009 16:48:01 GMT
Hi,

I'm setting up a small cluster with 2 machines. One is called 'master' and
one is called 'slave'. The master is the Hadoop master. I'm running Hadoop
0.20.0 and HBase from svn (the 0.20 branch).
On the master, I want to run the HBase master, and on the slave, I want to
run a regionserver and a Zookeeper instance.

hbase-site:

<property>
<name>hbase.rootdir</name>
<value>hdfs://master:9000/hbase</value>
<description>The directory shared by region servers.
</description>
</property>
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>3000</value>
</property>
<property>
<name>hbase.hregion.max.filesize</name>
<value>32000000</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>slave</value>
<description>Comma separated list of servers in the ZooKeeper Quorum.
For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".
By default this is set to localhost for local and pseudo-distributed modes
of operation. For a fully-distributed setup, this should be set to a full
list of ZooKeeper quorum servers. If HBASE_MANAGES_ZK is set in hbase-env.sh
this is the list of servers which we will start/stop ZooKeeper on.
</description>
</property>

Upon launching my job (on master), Zookeeper seems to crash (or something
like it).

I get the following output:
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.2.0--1, built on 05/15/2009 06:05 GMT
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Client environment:host.name
=master
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_14
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Client
environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.14/jre
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Client
environment:java.class.path=/root/installation/hadoop/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/root/installation/hadoop/bin/..:/root/installation/hadoop/bin/../hadoop-0.20.0-core.jar:/root/installation/hadoop/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/root/installation/hadoop/bin/../lib/commons-codec-1.3.jar:/root/installation/hadoop/bin/../lib/commons-el-1.0.jar:/root/installation/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:/root/installation/hadoop/bin/../lib/commons-logging-1.0.4.jar:/root/installation/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/root/installation/hadoop/bin/../lib/commons-net-1.4.1.jar:/root/installation/hadoop/bin/../lib/core-3.1.1.jar:/root/installation/hadoop/bin/../lib/hbase-0.20.0.jar:/root/installation/hadoop/bin/../lib/heritrix-1.14.3.jar:/root/installation/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/root/installation/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:/root/installation/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/root/installation/hadoop/bin/../lib/jets3t-0.6.1.jar:/root/installation/hadoop/bin/../lib/jetty-6.1.14.jar:/root/installation/hadoop/bin/../lib/jetty-util-6.1.14.jar:/root/installation/hadoop/bin/../lib/junit-3.8.1.jar:/root/installation/hadoop/bin/../lib/kfs-0.2.2.jar:/root/installation/hadoop/bin/../lib/log4j-1.2.15.jar:/root/installation/hadoop/bin/../lib/oro-2.0.8.jar:/root/installation/hadoop/bin/../lib/servlet-api-2.5-6.1.14.jar:/root/installation/hadoop/bin/../lib/slf4j-api-1.4.3.jar:/root/installation/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/root/installation/hadoop/bin/../lib/xmlenc-0.52.jar:/root/installation/hadoop/bin/../lib/zookeeper-r785019-hbase-1329.jar:/root/installation/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/root/installation/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Client
environment:java.library.path=/root/installation/hadoop/bin/../lib/native/Linux-i386-32
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Client environment:os.arch=i386
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Client
environment:os.version=2.6.24-6-xen
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Client environment:user.name
=root
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Client
environment:user.home=/root
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Client
environment:user.dir=/root/installation/hadoop
09/08/21 16:32:34 INFO zookeeper.ZooKeeper: Initiating client connection,
host=localhost:2181 sessionTimeout=30000
watcher=org.apache.hadoop.hbase.client.HConnectionManager$TableServers@126e85f
09/08/21 16:32:34 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset
is false
09/08/21 16:32:34 INFO zookeeper.ClientCnxn: Attempting connection to server
localhost/127.0.0.1:2181
09/08/21 16:32:34 INFO zookeeper.ClientCnxn: Priming connection to
java.nio.channels.SocketChannel[connected
local=/127.0.0.1:39093remote=localhost/
127.0.0.1:2181]
09/08/21 16:32:34 INFO zookeeper.ClientCnxn: Server connection successful
09/08/21 16:32:34 WARN zookeeper.ClientCnxn: Exception closing session 0x0
to sun.nio.ch.SelectionKeyImpl@af8358
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
    at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
09/08/21 16:32:34 WARN zookeeper.ClientCnxn: Ignoring exception during
shutdown input
java.net.SocketException: Transport endpoint is not connected
    at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
    at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
    at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
    at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
09/08/21 16:32:34 WARN zookeeper.ClientCnxn: Ignoring exception during
shutdown output
java.net.SocketException: Transport endpoint is not connected
    at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
    at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
    at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
    at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
09/08/21 16:32:34 INFO client.HConnectionManager$TableServers: getMaster
attempt 0 of 10 failed; retrying after sleep of 2000
java.io.IOException:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:331)
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:240)
    at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:257)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
    at com.acquia.web.Main.tryCreateDatabaseTables(Main.java:180)
    at com.acquia.web.Main.main(Main.java:60)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
    at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:750)
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:327)
    ... 10 more
09/08/21 16:32:36 INFO zookeeper.ClientCnxn: Attempting connection to server
localhost/127.0.0.1:2181
09/08/21 16:32:36 INFO zookeeper.ClientCnxn: Priming connection to
java.nio.channels.SocketChannel[connected
local=/127.0.0.1:39094remote=localhost/
127.0.0.1:2181]
09/08/21 16:32:36 INFO zookeeper.ClientCnxn: Server connection successful
09/08/21 16:32:36 WARN zookeeper.ClientCnxn: Exception closing session 0x0
to sun.nio.ch.SelectionKeyImpl@4a63d8
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
    at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
09/08/21 16:32:36 WARN zookeeper.ClientCnxn: Ignoring exception during
shutdown input
java.net.SocketException: Transport endpoint is not connected
    at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
    at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
    at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
    at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
09/08/21 16:32:36 WARN zookeeper.ClientCnxn: Ignoring exception during
shutdown output
java.net.SocketException: Transport endpoint is not connected
    at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
    at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
    at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
    at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
09/08/21 16:32:37 INFO zookeeper.ClientCnxn: Attempting connection to server
localhost/127.0.0.1:2181
09/08/21 16:32:37 INFO zookeeper.ClientCnxn: Priming connection to
java.nio.channels.SocketChannel[connected
local=/127.0.0.1:39095remote=localhost/
127.0.0.1:2181]
09/08/21 16:32:37 INFO zookeeper.ClientCnxn: Server connection successful
09/08/21 16:32:37 WARN zookeeper.ClientCnxn: Exception closing session 0x0
to sun.nio.ch.SelectionKeyImpl@1e0ff2f
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0
lim=4 cap=4]
    at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:653)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:897)
09/08/21 16:32:37 WARN zookeeper.ClientCnxn: Ignoring exception during
shutdown input
java.net.SocketException: Transport endpoint is not connected
    at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
    at
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
    at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
    at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:951)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
09/08/21 16:32:37 WARN zookeeper.ClientCnxn: Ignoring exception during
shutdown output
java.net.SocketException: Transport endpoint is not connected
    at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
    at
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
    at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
    at
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:956)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:922)
09/08/21 16:32:37 INFO client.HConnectionManager$TableServers: getMaster
attempt 1 of 10 failed; retrying after sleep of 2000
java.io.IOException:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:331)
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:240)
    at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:257)
    at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
    at com.acquia.web.Main.tryCreateDatabaseTables(Main.java:180)
    at com.acquia.web.Main.main(Main.java:60)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
    at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:750)
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:327)
    ... 10 more



I'm quite certain Zookeeper is in fact running at this point in time, and
the Zookeeper logs look fine. I think (looking at the logs) the master might
be trying to connect to localhost instead of to slave, but I can't find a
reason why.

Thanks in advance!
Mathias

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message