hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Hbase stops working
Date Thu, 26 Feb 2009 13:41:05 GMT
Andrew,

Oh yes sorry, your setup is in local mode so no region server in a different
thread. My bad!

That lease timeout probably means that you were swapping and a full jvm
garbage collecting occurred which blocked the main thread. Can you make sure
that it did swap at that moment?

It would also be helpful to get the usual info like hardware, OS, hbase
version, etc.

Thx!

J-D

On Thu, Feb 26, 2009 at 8:35 AM, Andrew McCall <andrew.mccall@goroam.net>wrote:

> Hi J-D,
>
> I did check the region server logs and they didn't seem to be very useful
> at all:
>
> Mon Feb 23 22:11:57 GMT 2009 Starting regionserver on betty.goroam.net
> ulimit -n 1024
> 2009-02-23 22:11:58,817 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Not starting a distinct
> region server because hbase.master is set to 'local' mode
>
> Is all that is in them, once for each time I started HBase.
>
> I saw the shutdown message, which was preceded by this:
>
> 2009-02-24 09:34:41,579 INFO org.apache.hadoop.hbase.master.ServerManager:
> 127.0.0.1:40076 lease expired
>
> I can't find any evidence it's related to my map/reduce task, it's the only
> one in the log files and it's suspiciously just before the process shutdown
> message - but, I'm new to this and honestly don't know what to make of it.
>
> Andrew
>
>
> On 26 Feb 2009, at 12:57, Jean-Daniel Cryans wrote:
>
>  Andrew,
>>
>> For some reason, your region server shut down around 9:36:
>>
>> 2009-02-24 09:36:05,884 INFO org.apache.hadoop.hbase.
>>
>>>
>>> master.RegionServerOperation: process shutdown of server 127.0.0.1:40076
>>> :
>>> logSplit: false, rootRescanned: false, numberOfMetaRegions: 1,
>>>
>>
>>
>> But just before that the master received a startup message from it. Did
>> you
>> take a look at your region server log? What does it say around that time?
>>
>> Thx
>>
>> J-D
>>
>> On Thu, Feb 26, 2009 at 5:34 AM, Andrew McCall <andrew.mccall@goroam.net
>> >wrote:
>>
>>  I've been testing a nuch on HBase and I'm running into an issue with it
>>> when it's been running for a while.
>>>
>>> I'm running hbase 0.19 on a single local host pseudo distributed hadoop
>>> 0.19 cluster.
>>>
>>> After some time I get the following in my logs:
>>>
>>> 2009-02-24 09:31:18,664 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> All 1 .META. region(s) scanned
>>> 2009-02-24 09:31:18,664 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>>> startKey: <>, server: 127.0.0.1:40076}
>>> 2009-02-24 09:31:24,293 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
>>> 076} complete
>>> 2009-02-24 09:31:24,293 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
>>> startKey: <>, server: 127.0.0.1:40076}
>>> 2009-02-24 09:31:27,217 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname:
>>> .META.,,1, startKey: <>, server: 127.0.0.1:40
>>> 076} complete
>>> 2009-02-24 09:31:27,217 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> All 1 .META. region(s) scanned
>>> 2009-02-24 09:31:55,533 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
>>> 076} complete
>>> 2009-02-24 09:32:18,665 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>>> startKey: <>, server: 127.0.0.1:40076}
>>> 2009-02-24 09:32:35,749 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
>>> startKey: <>, server: 127.0.0.1:40076}
>>> 2009-02-24 09:32:38,957 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname:
>>> .META.,,1, startKey: <>, server: 127.0.0.1:40
>>> 076} complete
>>> 2009-02-24 09:32:38,957 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> All 1 .META. region(s) scanned
>>> 2009-02-24 09:33:18,771 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>>> startKey: <>, server: 127.0.0.1:40076}
>>> 2009-02-24 09:33:54,889 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
>>> 076} complete
>>> 2009-02-24 09:33:57,387 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0,
>>> startKey: <>, server: 127.0.0.1:40076}
>>> 2009-02-24 09:34:41,579 INFO
>>> org.apache.hadoop.hbase.master.ServerManager:
>>> 127.0.0.1:40076 lease expired
>>> 2009-02-24 09:34:53,440 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname:
>>> .META.,,1, startKey: <>, server: 127.0.0.1:40
>>> 076} complete
>>> 2009-02-24 09:34:53,441 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> All 1 .META. region(s) scanned
>>> 2009-02-24 09:34:53,441 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>>> startKey: <>, server: 127.0.0.1:40076}
>>> 2009-02-24 09:35:38,720 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname:
>>> .META.,,1, startKey: <>, server: 127.0.0.1:40
>>> 076} complete
>>> 2009-02-24 09:35:38,720 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> All 1 .META. region(s) scanned
>>> 2009-02-24 09:35:56,951 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>>> startKey: <>, server: 127.0.0.1:40076}
>>> 2009-02-24 09:36:05,667 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegionServer:
>>> MSG_CALL_SERVER_STARTUP
>>> 2009-02-24 09:36:05,668 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.rootScanner scan of 1 row(s) of meta region {regionname:
>>> -ROOT-,,0, startKey: <>, server: 127.0.0.1:40
>>> 076} complete
>>> 2009-02-24 09:36:05,884 INFO
>>> org.apache.hadoop.hbase.master.RegionServerOperation: process shutdown of
>>> server 127.0.0.1:40076: logSplit: false, rootRescanned: false,
>>> numberOfMetaRegions: 1,
>>> onlineMetaRegions.size(): 1
>>> 2009-02-24 09:36:17,689 INFO org.apache.hadoop.hbase.regionserver.HLog:
>>> Splitting 1 log(s) in
>>> hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076
>>> 2009-02-24 09:36:25,034 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scan of 2 row(s) of meta region {regionname:
>>> .META.,,1, startKey: <>, server: 127.0.0.1:40
>>> 076} complete
>>> 2009-02-24 09:36:25,034 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> All 1 .META. region(s) scanned
>>> 2009-02-24 09:36:36,598 INFO org.apache.hadoop.hbase.regionserver.HLog:
>>> log
>>> file splitting completed for
>>> hdfs://localhost:9000/hbase/log_127.0.0.1_1235427119323_40076
>>> 2009-02-24 09:36:56,953 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>>> startKey: <>, server: 127.0.0.1:40076}
>>> 2009-02-24 09:37:05,279 ERROR
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>>      at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>>      at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>>>      at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>>      at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>      at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>>      at
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
>>> 2009-02-24 09:37:06,102 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>> Server
>>> handler 4 on 40076, call openScanner([B@1ac5a78, [[B@6799dc, [B@1a66cfc,
>>> 9223372036854775807, null) from 127.0.0.1
>>> :50139: error: org.apache.hadoop.hbase.NotServingRegionException:
>>> .META.,,1
>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>>      at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>>      at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>>>      at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>>      at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>      at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>>      at
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
>>> 2009-02-24 09:37:07,051 WARN org.apache.hadoop.hbase.master.BaseScanner:
>>> Scan one META region: {regionname: .META.,,1, startKey: <>, server:
>>> 127.0.0.1:40076}
>>> org.apache.hadoop.hbase.NotServingRegionException:
>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>>      at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>>      at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>>>      at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>>      at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>      at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>>      at
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
>>>
>>>      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>>      at
>>>
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>>      at
>>>
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>      at
>>>
>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:95)
>>>      at
>>>
>>> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185)
>>>      at
>>>
>>> org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
>>>      at
>>>
>>> org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
>>>      at
>>> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
>>>      at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
>>> 2009-02-24 09:37:07,230 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> All 1 .META. region(s) scanned
>>> 2009-02-24 09:37:50,642 INFO
>>> org.apache.hadoop.hbase.regionserver.HRegion:
>>> Closed .META.,,1
>>> 2009-02-24 09:37:56,955 INFO org.apache.hadoop.hbase.master.BaseScanner:
>>> RegionManager.metaScanner scanning meta region {regionname: .META.,,1,
>>> startKey: <>, server: 127.0.0.1:40076}
>>> 2009-02-24 09:37:57,126 ERROR
>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed openScanner
>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>>      at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>>      at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>>>      at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>>      at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>      at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>>      at
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
>>> 2009-02-24 09:37:59,129 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>>> Server
>>> handler 2 on 40076, call openScanner([B@155bf4e, [[B@82b436, [B@1056757,
>>> 9223372036854775807, null) from 127.0.0.1
>>> :35804: error: org.apache.hadoop.hbase.NotServingRegionException:
>>> .META.,,1
>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>>      at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>>      at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>>>      at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>>      at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>      at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>>      at
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
>>> 2009-02-24 09:38:00,412 WARN org.apache.hadoop.hbase.master.BaseScanner:
>>> Scan one META region: {regionname: .META.,,1, startKey: <>, server:
>>> 127.0.0.1:40076}
>>> org.apache.hadoop.hbase.NotServingRegionException:
>>> org.apache.hadoop.hbase.NotServingRegionException: .META.,,1
>>>      at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
>>>      at
>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1699)
>>>      at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>>>      at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>      at java.lang.reflect.Method.invoke(Method.java:597)
>>>      at
>>> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
>>>      at
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)
>>>
>>>      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>>      at
>>>
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>>>      at
>>>
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>>      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>>      at
>>>
>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:95)
>>>      at
>>>
>>> org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:185)
>>>      at
>>>
>>> org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
>>>      at
>>>
>>> org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
>>>      at
>>> org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:137)
>>>      at org.apache.hadoop.hbase.Chore.run(Chore.java:65)
>>>
>>> This just repeats forever, until I restart HBase.
>>>
>>> Any help would be greatly appreciated.
>>> Andrew
>>>
>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message