hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dimitri Goldin <dimitri.gol...@neofonie.de>
Subject Eternal RIT problem when RS tries to access wrong region-folder on HDFS
Date Thu, 02 May 2013 14:17:47 GMT
Hi,

I have a strange RIT problem with a single region of our biggest table.
After an hbck (wondering why it only discovered it at that time) it
started trying to assign a region which has been bouncing between
OFFLINE/PENDING_OPEN/OPENING for two days.

I already tried close_region/unassign with force and even the good-old
delete /hbase node in zookeeper, but we still experience the same issue.

Interestinly, the full regions id is
'documents,7128586022887322720,1363696791400.79c619508659018ff3ef0887611eb8f7.' 

but in the exception the filename it tries to open is:
'/hbase/documents/5b9c16898a371de58f31f0bdf86b1f8b/d/0707b1ec4c6b41cf9174e0d2a1785fe9'. 


Rough sequence from the logs seems to be the following:

===
* Received request to open region:
documents,7128586022887322720,1363696791400.79c619508659018ff3ef0887611eb8f7.

* Setting up tabledescriptor config now ...

* Opening of region {NAME =>
'documents,7128586022887322720,1363696791400.79c619508659018ff3ef0887611eb8f7.',
     STARTKEY => '7128586022887322720',
     ENDKEY => '7130716361635801616',
     ENCODED => 79c619508659018ff3ef0887611eb8f7,} failed, marking as 
FAILED_OPEN in ZK

* File does not exist:
 
/hbase/documents/5b9c16898a371de58f31f0bdf86b1f8b/d/0707b1ec4c6b41cf9174e0d2a1785fe9 
[...]
===

As the Exception implies, 
'/hbase/documents/5b9c16898a371de58f31f0bdf86b1f8b' does not exist,
while the '/hbase/documents/79c619508659018ff3ef0887611eb8f7' folder 
exists and contains all necessary files.

I've checked .META. thinking that the regions ENCODED field might
be broken, which is not the case judging by the 3rd. log-message.
Otherwise, I'm out of ideas how the encoded-region part might get
switched with another value.

Any ideas what might cause such a behaviour and how to fix it?

HBase version: 0.92.1-cdh4.1.2

Complete log-message including stacktrace of the FileNotFound
Exception: http://fpaste.org/10005/04104136/ (Sorry for the format)


Thanks in advance,
     Dimitry

-- 
----------------------------------
Dimitry Goldin
Software Developer

Neofonie GmbH
Robert-Koch-Platz 4
10115 Berlin

T: +49 30 246 27 413

goldin@neofonie.de <mailto:goldin@neofonie.de>
http://www.neofonie.de

Handelsregister
Berlin-Charlottenburg: HRB 67460

Geschäftsführung:
Thomas Kitlitschko

Mime
View raw message