hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
Date Mon, 17 Nov 2014 20:15:32 GMT
Seems to be a zookeeper setup issue.

Mind pastebin'ing your config (for 3 zookeeper servers) ?

Please also check zookeeper server log.

Cheers

On Mon, Nov 17, 2014 at 11:58 AM, eluiggi <eduardoluiggi@gmail.com> wrote:

> I have tried that as is one of the suggestions from Cloudera manager.
> However, adding the servers results in none of them able to talk to
> zookeeper (not even the one on the sharing the same node) and therefore
> Hbase completely down. The master throws an exception related to the one
> thrown by the region servers.
>
> 2014-11-17 14:50:20,590 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket
> connection to server ip-10-146-188-157.ec2.internal/10.146.188.157:2181.
> Will not attempt to authenticate using SASL (unknown error)
> 2014-11-17 14:50:20,591 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to
> ip-10-146-188-157.ec2.internal/10.146.188.157:2181, initiating session
> 2014-11-17 14:50:20,592 INFO org.apache.zookeeper.ClientCnxn: Unable to
> read
> additional data from server sessionid 0x0, likely server has closed socket,
> closing socket connection and attempting reconnect
> 2014-11-17 14:50:22,576 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket
> connection to server ip-10-164-167-107.ec2.internal/10.164.167.107:2181.
> Will not attempt to authenticate using SASL (unknown error)
> 2014-11-17 14:51:00,726 INFO org.apache.zookeeper.ClientCnxn: Client
> session
> timed out, have not heard from server in 40032ms for sessionid 0x0, closing
> socket connection and attempting reconnect
> 2014-11-17 14:51:00,826 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
> ZooKeeper,
>
> quorum=ip-10-146-194-138.ec2.internal:2181,ip-10-146-188-157.ec2.internal:2181,ip-10-164-167-107.ec2.internal:2181,
> exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
> 2014-11-17 14:51:00,827 ERROR
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper create
> failed after 4 attempts
> 2014-11-17 14:51:00,828 ERROR
> org.apache.hadoop.hbase.master.HMasterCommandLine: Master exiting
> java.lang.RuntimeException: Failed construction of Master: class
> org.apache.hadoop.hbase.master.HMaster
>         at
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2775)
>         at
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:184)
>         at
>
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:134)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>         at
>
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
>         at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2789)
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>         at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
>         at
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:489)
>         at
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:468)
>         at
>
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1233)
>         at
>
> org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1211)
>         at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:174)
>         at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:167)
>         at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:472)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>         at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>         at
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2770)
>         ... 5 more
>
> One other test that I made was to connect to the zookeeper from one of the
> region server nodes using zkCli.sh. It looks like the connection is
> established but sockets are closed and reopen constantly as the timeout
> limit is reached.
>
> Thanks for the help!
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/ConnectionLossException-KeeperErrorCode-ConnectionLoss-for-hbase-master-tp4066034p4066039.html
> Sent from the HBase User mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message