hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dru Jensen <drujen...@gmail.com>
Subject Re: rowcounter fails - Region closed
Date Thu, 20 Nov 2008 20:36:51 GMT
Thanks Stack.  Yes, upping the xceivercount fixed the previous issue.   
The system is more stable than ever. I used the "spelling" you  
provided. Maybe add to FAQ?

grep the master I see the region but unfortunately, I did not have  
DEBUG on at the time.  I added DEBUG to the log4j settings.  After  
restarting I saw this:

hbase-root-master-dev-hadoop1.brandaffinity.net.log:2008-11-20  
10:28:31,265 DEBUG org.apache.hadoop.hbase.master.BaseScanner: Current  
assignment of metrics,Game Face|News,1226620888277 is not valid:  
serverInfo: address: 10.10.10.51:60020, startcode: 1227205702706,  
load: (requests: 4 regions: 1), passed startCode: 1227124620094,  
storedInfo.startCode: 1227205702706, unassignedRegions: false,  
pendingRegions: false

hbase-root-master-dev-hadoop1.brandaffinity.net.log:2008-11-20  
10:28:48,156 INFO org.apache.hadoop.hbase.master.RegionManager:  
assigning region metrics,Game Face|News,1226620888277 to server  
10.10.10.51:60020

I ran the jobcounter again and it was successful.  Looks like HBase  
recovered on it's own.

Thanks again,
Dru


On Nov 20, 2008, at 10:11 AM, stack wrote:

> Says its 'closed'.
>
> Look in master log for references to this region.  See if it was  
> ever online.  If it was, see which host it was up on; e.g. do  
> something like "grep 'metrics,Game Face|News,1226620888277'"  
> master.log".  Look at its RegionInfo -- should be output in master  
> log if you have DEBUG enabled.  Look at the encoded name.   See that  
> the region exists still in hdfs by doing "./bin/hadoop fs -lsr  
> HBASE_ROOTDIR/metrics/ENCODED_NAME.   Next add '-v metaScanner' to  
> weed the metaScanner output when grepping master log to see trail of  
> events as they pertain to this particular region.  Might give you a  
> clue as to what happened.  When you scan the .META., does this  
> region appear at all?  Is there a 'hole' in the .META. where this  
> region should be?  We may have to reinsert if so.  To do this, will  
> need old value for HRI.
>
> Did the upping of xceivercount work for your previous issue Dru?
>
> St.Ack
>
>
>
> Dru Jensen wrote:
>> I have a table that has 20+ million rows.  I tried to run  
>> rowcounter MR process against it but one of the task attempts fail  
>> on the following exception:
>>
>> java.io.IOException: java.io.IOException: Region metrics,Game Face| 
>> News,1226620888277 closed
>>    at  
>> org 
>> .apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java: 
>> 1336)
>>    at  
>> org 
>> .apache 
>> .hadoop 
>> .hbase.regionserver.HRegionServer.openScanner(HRegionServer.java: 
>> 1175)
>>    at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
>>    at  
>> sun 
>> .reflect 
>> .DelegatingMethodAccessorImpl 
>> .invoke(DelegatingMethodAccessorImpl.java:25)
>>    at java.lang.reflect.Method.invoke(Method.java:585)
>>    at org.apache.hadoop.hbase.ipc.HbaseRPC 
>> $Server.call(HbaseRPC.java:554)
>>    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
>>
>> I checked .meta. and I don't see this region online.  Should it be?  
>> Any ideas how to fix it?
>>
>> Thanks,
>> Dru
>>
>>
>


Mime
View raw message