hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stack <st...@duboce.net>
Subject Re: Issue with NotServingRegionException
Date Tue, 05 Aug 2008 20:54:08 GMT
Tim Sell wrote:
> ...
> I understand until recently the logs were filled with warnings about
> this during every split while a region was temporarily offline so to
> speak? it would figure itself out once the region was opened, but
> logging these warnings was removed because it was confusing people?
No.  They are still there.  They are logged at DEBUG level unless we 
exhaust retries.  Then you'll get a NSRE out on your client as an error 
(That said, they are showing in your logs here because they are logged 
at INFO level by hadoop ipc).

> This isn't one of those warnings, it's the only
> NotServingRegionException in my logs.
> I'm using this morning's trunk.
> I've attached my master and region server logs, if anyone can help. It
> happens around the 2008-08-05 12:06 mark.
> I added some blank lines to help you find where the activity stopped.
> (there is about a 30 minute gap in the region server logs afterwards
> as there were no more input to be compacted, split, etc).
I took a look.  Seems fine.

As Andrew asks, the thrift client log would be handy....

But this in your second message is disturbing:

08/08/05 13:56:38 DEBUG client.HConnectionManager$TableServers:
reloading table servers because: HRegionInfo was null or empty in

We're stuck trying to get .META. info but some rows are damaged; have 
null content.

For sure you are at TRUNK from this morning?  I thought we printed out 
all that was non-empty from the row to give us a clue which row is busted.

What happens if you try and scan using the shell?

 > scan '.META.'


> [Towards the end there is some irrelevant stuff logged because I was
> fiddling in the hbase shell. ignore that]
> ~Tim.
> 2008/8/5 Andrew Purtell <apurtell@yahoo.com>:
>> Hello Nickolai,
>> When your regions unexpectedly closed, is/was there anything
>> logged to the regionserver and master logs at those times?
>> As for your problem right now, please consider stopping, moving
>> the old log files out of the way, and then restarting. Please
>> then try to access your table. Then, please send the resulting
>> master and region server logs. I'll be happy to take a look.
>> Thanks,
>>    - Andy
>>> From: Nickolai Toupikov <nickolai.toupikov@deri.org>
>>> Subject: Issue with NotServingRegionException
>>> To: hbase-user@hadoop.apache.org
>>> Date: Tuesday, August 5, 2008, 5:47 AM
>>> Hello!
>>> It seems I have a serious problem with HBase. We've
>>> been using if for some time now, had some minor problems with
>>> regions unexpectedly closing once every couple of days, but
>>> this would usually sort itself out after a couple of stops and
>>> starts.
> >

View raw message