hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ch huang <justlo...@gmail.com>
Subject Re: master node fail to start
Date Wed, 10 Jul 2013 01:30:38 GMT
i reinstalled hbase , and how the error info is different



13/07/10 09:17:52 WARN conf.Configuration: fs.default.name is deprecated.
Instead, use fs.defaultFS
13/07/10 09:17:54 INFO master.ServerManager: Finished waiting for region
servers count to settle; checked in 1, slept for 12831 ms, expecting
minimum of 1, maximum of 2147483647
, master is running.
13/07/10 09:17:54 INFO master.MasterFileSystem: Log folder
hdfs://CH22:9000/hbaseroot/.logs/CH34,60020,1373419072288 belongs to an
existing region server
13/07/10 09:17:54 INFO master.MasterFileSystem: No logs to split
13/07/10 09:17:54 FATAL master.HMaster: Master server abort: loaded
coprocessors are: []
13/07/10 09:17:54 FATAL master.HMaster: Unhandled exception. Starting
shutdown.
java.lang.IllegalArgumentException: offset (0) + length (2) exceed the
capacity of the array: 0
        at
org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:516)
        at org.apache.hadoop.hbase.util.Bytes.toShort(Bytes.java:738)
        at org.apache.hadoop.hbase.util.Bytes.toShort(Bytes.java:714)
        at
org.apache.hadoop.hbase.ServerName.parseVersionedServerName(ServerName.java:276)
        at
org.apache.hadoop.hbase.executor.RegionTransitionData.readFields(RegionTransitionData.java:191)
        at
org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:133)
        at
org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:103)
        at
org.apache.hadoop.hbase.executor.RegionTransitionData.fromBytes(RegionTransitionData.java:238)
        at
org.apache.hadoop.hbase.zookeeper.ZKAssign.getDataAndWatch(ZKAssign.java:882)
        at
org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:518)
        at
org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransitionAndBlockUntilAssigned(AssignmentManager.java:489)
        at
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:679)
        at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:583)
        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:395)
        at java.lang.Thread.run(Thread.java:662)
13/07/10 09:17:54 INFO master.HMaster: Aborting
13/07/10 09:17:54 INFO ipc.HBaseServer: Stopping server on 60000
13/07/10 09:17:54 INFO ipc.HBaseServer: IPC Server handler 0 on 60000:
exiting
13/07/10 09:17:54 INFO ipc.HBaseServer: REPL IPC Server handler 1 on 60000:
exiting
13/07/10 09:17:54 INFO ipc.HBaseServer: IPC Server handler 5 on 60000:
exiting


On Wed, Jul 10, 2013 at 8:54 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Time range don't match.
>
> Can you find log around 2013-07-09 15:47:09,067 ?
>
> Cheers
>
> On Tue, Jul 9, 2013 at 5:52 PM, ch huang <justlooks@gmail.com> wrote:
>
> > here is CH35 region server out log
> >
> > 13/07/10 08:23:34 INFO regionserver.HRegionServer: Serving as
> > CH35,60020,1373364396982, RPC listening on CH35/192.168.10.35:60020,
> > sessionid=0x3fc29ea7490009
> > 13/07/10 08:23:34 INFO regionserver.SplitLogWorker: SplitLogWorker
> > CH35,60020,1373364396982 starting
> > 13/07/10 08:23:34 INFO regionserver.HRegionServer: Registered
> RegionServer
> > MXBean
> > 13/07/10 08:23:43 INFO util.ChecksumType: Checksum using
> > org.apache.hadoop.util.PureJavaCrc32
> > 13/07/10 08:23:43 INFO util.ChecksumType: Checksum can use
> > org.apache.hadoop.util.PureJavaCrc32C
> > 13/07/10 08:37:28 INFO regionserver.HRegionServer: Attempting connect to
> > Master server at CH22,60000,1373416645157
> > 13/07/10 08:37:28 INFO regionserver.HRegionServer: Connected to master at
> > CH22/192.168.10.22:60000
> > 13/07/10 08:45:31 INFO zookeeper.ClientCnxn: Unable to read additional
> data
> > from server sessionid 0x3fc29ea7490009, likely server has closed socket,
> > closing socket connection and attempting reconnect
> > 13/07/10 08:45:33 INFO zookeeper.ClientCnxn: Opening socket connection to
> > server CH22/192.168.10.22:2281. Will not attempt to authenticate using
> > SASL
> > (Unable to locate a login configuration)
> > 13/07/10 08:45:33 WARN zookeeper.ClientCnxn: Session 0x3fc29ea7490009 for
> > server null, unexpected error, closing socket connection and attempting
> > reconnect
> > java.net.ConnectException: Connection refused
> >         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >         at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> >         at
> >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> >         at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> > 13/07/10 08:45:35 INFO zookeeper.ClientCnxn: Opening socket connection to
> > server CH22/192.168.10.22:2281. Will not attempt to authenticate using
> > SASL
> > (Unable to locate a login configuration)
> > 13/07/10 08:45:35 INFO zookeeper.ClientCnxn: Socket connection
> established
> > to CH22/192.168.10.22:2281, initiating session
> > 13/07/10 08:45:35 INFO zookeeper.ClientCnxn: Unable to read additional
> data
> > from server sessionid 0x3fc29ea7490009, likely server has closed socket,
> > closing socket connection and attempting reconnect
> > 13/07/10 08:45:37 INFO zookeeper.ClientCnxn: Opening socket connection to
> > server CH22/192.168.10.22:2281. Will not attempt to authenticate using
> > SASL
> > (Unable to locate a login configuration)
> > 13/07/10 08:45:37 INFO zookeeper.ClientCnxn: Socket connection
> established
> > to CH22/192.168.10.22:2281, initiating session
> > 13/07/10 08:45:37 INFO zookeeper.ClientCnxn: Session establishment
> complete
> > on server CH22/192.168.10.22:2281, sessionid = 0x3fc29ea7490009,
> > negotiated
> > timeout = 40000
> > 13/07/10 08:48:28 INFO zookeeper.ClientCnxn: Unable to read additional
> data
> > from server sessionid 0x3fc29ea7490009, likely server has closed socket,
> > closing socket connection and attempting reconnect
> > 13/07/10 08:48:30 INFO zookeeper.ClientCnxn: Opening socket connection to
> > server CH22/192.168.10.22:2281. Will not attempt to authenticate using
> > SASL
> > (Unable to locate a login configuration)
> > 13/07/10 08:48:30 WARN zookeeper.ClientCnxn: Session 0x3fc29ea7490009 for
> > server null, unexpected error, closing socket connection and attempting
> > reconnect
> > java.net.ConnectException: Connection refused
> >         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >         at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> >         at
> >
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> >         at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> > 13/07/10 08:48:31 INFO zookeeper.ClientCnxn: Opening socket connection to
> > server CH22/192.168.10.22:2281. Will not attempt to authenticate using
> > SASL
> > (Unable to locate a login configuration)
> >
> >
> > On Wed, Jul 10, 2013 at 8:37 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of
> > > -ROOT-,,0.70236052 to serverName=CH35,60020,1372991820903,
> > >
> > > Have you checked region server log for CH35 ?
> > >
> > > On Tue, Jul 9, 2013 at 5:35 PM, ch huang <justlooks@gmail.com> wrote:
> > >
> > > > i upgrade cdh3u4 to cdh4.3,start master node have problem
> > > >
> > > >
> > > > 2013-07-09 15:47:09,061 INFO
> > > > org.apache.hadoop.hbase.catalog.RootLocationEditor: Unsetting ROOT
> > region
> > > > location in ZooKeeper
> > > > 2013-07-09 15:47:09,063 DEBUG
> > org.apache.hadoop.hbase.zookeeper.ZKAssign:
> > > > master:60000-0x3fa281450100bd Creating (or updating) unassigned node
> > for
> > > > 70236052 with OFFLINE state
> > > > 2013-07-09 15:47:09,066 INFO
> > > > org.apache.hadoop.hbase.master.AssignmentManager: No previous
> > transition
> > > > plan was found (or we are ignoring an existing plan) for
> > > -ROOT-,,0.70236052
> > > > so generated a random one; hri=-ROOT-,,0.70236052, src=,
> > > > dest=CH35,60020,1372991820903; 1 (online=1, exclude=null) available
> > > servers
> > > > 2013-07-09 15:47:09,066 INFO
> > > > org.apache.hadoop.hbase.master.AssignmentManager: Assigning region
> > > > -ROOT-,,0.70236052 to CH35,60020,1372991820903
> > > > 2013-07-09 15:47:09,067 WARN
> > > > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment
> of
> > > > -ROOT-,,0.70236052 to serverName=CH35,60020,1372991820903,
> > > > load=(requests=0, regions=79, usedHeap=6079, maxHeap=19443), trying
> to
> > > > assign elsewhere instead; retry=0
> > > > java.net.ConnectException: Connection refused
> > > >         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> > > >         at
> > > >
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> > > >         at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:429)
> > > >         at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:394)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
> > > >         at
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
> > > >         at
> > > > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
> > > >         at
> > > >
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
> > > >         at $Proxy7.openRegion(Unknown Source)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:573)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1127)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:912)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:892)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.AssignmentManager.assignRoot(AssignmentManager.java:1396)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.verifyAndAssignRoot(ServerShutdownHandler.java:106)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.verifyAndAssignRootWithRetries(ServerShutdownHandler.java:124)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:183)
> > > >         at
> > > >
> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:163)
> > > >         at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > > >         at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > > >         at java.lang.Thread.run(Thread.java:662)
> > > > 2013-07-09 15:47:09,068 WARN
> > > > org.apache.hadoop.hbase.master.AssignmentManager: Unable to find a
> > viable
> > > > location to assign region -ROOT-,,0.70236052
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message