hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: The problems of power outages
Date Wed, 15 Jan 2014 03:26:49 GMT
I went back from 01-13 all the way
to hbase-lilaifeng-master-CentOS.log.2014-01-07
I saw:

2014-01-07 17:46:05,700 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Cached location for .META.,,1.1028785192 is Slave2:60020

Do you have region server log for Slave2 ?


On Tue, Jan 14, 2014 at 3:56 PM, 李来锋 <wlilaifeng@126.com> wrote:

> Hi,
>  Since the blackout caused part of the table can not be accessed, use
> "hbase HBCK - fix" an error occurred when repair:
> 14/01/14 23:12:59 DEBUG
> client.HConnectionManager$HConnectionImplementation: Removed
> .META.,,1.1028785192 for tableName=.META. from cache because of
> 14/01/14 23:12:59 DEBUG
> client.HConnectionManager$HConnectionImplementation: Cached location for
> .META.,,1.1028785192 is Slave2:60020
> Exception in thread "main"
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=10, exceptions:
> Tue Jan 14 22:59:55 CST 2014,
> org.apache.hadoop.hbase.client.ScannerCallable@51a19458,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1
> Tue Jan 14 23:00:15 CST 2014,
> org.apache.hadoop.hbase.client.ScannerCallable@51a19458,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1
> Tue Jan 14 23:00:35 CST 2014,
> org.apache.hadoop.hbase.client.ScannerCallable@51a19458,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1
> Tue Jan 14 23:00:56 CST 2014,
> org.apache.hadoop.hbase.client.ScannerCallable@51a19458,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1
> Tue Jan 14 23:01:36 CST 2014,
> org.apache.hadoop.hbase.client.ScannerCallable@51a19458,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1
> Tue Jan 14 23:02:16 CST 2014,
> org.apache.hadoop.hbase.client.ScannerCallable@51a19458,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1
> Tue Jan 14 23:03:37 CST 2014,
> org.apache.hadoop.hbase.client.ScannerCallable@51a19458,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1
> Tue Jan 14 23:04:57 CST 2014,
> org.apache.hadoop.hbase.client.ScannerCallable@51a19458,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1
> Tue Jan 14 23:07:38 CST 2014,
> org.apache.hadoop.hbase.client.ScannerCallable@51a19458,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1
> Tue Jan 14 23:12:59 CST 2014,
> org.apache.hadoop.hbase.client.ScannerCallable@51a19458,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Re online: .META.,,1
>  at
> org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183)
>  at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:206)
>  at
> org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:54)
>  at
> org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133)
>  at
> org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130)
>  at
> org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:383)
>  at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:130)
>  at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:105)
>  at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:83)
>  at
> org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:67)
>  at
> org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntries(HBaseFsck.java:2584)
>  at
> org.apache.hadoop.hbase.util.HBaseFsck.onlineConsistencyRepair(HBaseFsck.java:384)
>  at org.apache.hadoop.hbase.util.HBaseFsck.onlineHbck(HBaseFsck.java:437)
>  at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:3633)
>  at org.apache.hadoop.hbase.util.HBaseFsck.run(HBaseFsck.java:3452)
>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>  at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3446)
>
> This is the error log when the power outage:
> 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=2 regions=1
> average=0.5 mostloaded=1 leastloaded=0
> 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=2 regions=1
> average=0.5 mostloaded=1 leastloaded=0
> 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=2 regions=1
> average=0.5 mostloaded=1 leastloaded=0
> 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=2 regions=1
> average=0.5 mostloaded=1 leastloaded=0
> 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=2 regions=1
> average=0.5 mostloaded=1 leastloaded=0
> 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=2 regions=1
> average=0.5 mostloaded=1 leastloaded=0
> 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=2 regions=1
> average=0.5 mostloaded=1 leastloaded=0
> 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=2 regions=1
> average=0.5 mostloaded=1 leastloaded=0
> 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=2 regions=1
> average=0.5 mostloaded=1 leastloaded=0
> 2014-01-13 09:26:10,899 INFO org.apache.hadoop.hbase.master.LoadBalancer:
> Skipping load balancing because balanced cluster; servers=2 regions=1
> average=0.5 mostloaded=1 leastloaded=0
> 2014-01-13 09:26:10,923 DEBUG
> org.apache.hadoop.hbase.client.ClientScanner: Creating scanner over .META.
> starting at key ''
> 2014-01-13 09:26:10,923 DEBUG
> org.apache.hadoop.hbase.client.ClientScanner: Advancing internal scanner to
> startKey at ''
> 2014-01-13 09:26:10,929 DEBUG
> org.apache.hadoop.hbase.client.ClientScanner: Finished with scanning at
> {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}
> 2014-01-13 09:26:10,929 DEBUG
> org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 10 catalog row(s)
> and gc'd 0 unreferenced parent region(s)
>
> Attachment is the log files.
>
> Thanks
>
>                                    致
> 礼
>
>                          李来锋
>                          北京中恒博瑞数字电力科技有限公司
>                          邮编:  100085
>                          地址:  北京市海淀区西二旗大街39号领秀新硅谷D区B座
>                          TEL:  010-82781699-8181,18612035758
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message