hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From schubert zhang <zson...@gmail.com>
Subject Metadata and region mismatch
Date Thu, 12 Mar 2009 11:09:58 GMT
Hi all,
Today, I encounter a new issue about failure to batchUpdate commit.

I am running a program to insert rows into a HBase table, but after long
time of batchUpdating, following exception occur:

org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server Some server for region
TESTTABLE,13575565132@2008-12-0117:16:55.117,1236847258901, row
'13575581009@2008-12-0606:15:48.077', but failed after 10 attempts.
Exceptions:
        at
org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:942)
        at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1372)
        at org.apache.hadoop.hbase.client.HTable.close(HTable.java:1385)
        ......

And after waiting for a long time, I still cannot insert new data.

Then, I check the HBase status, all master and regionservers are running.

But, I find a mismatch about region
"TESTTABLE,13575565132@2008-12-0117:16:55.117,1236847258901".
In the metadata, I found it said this region is severed by 10.24.1.12, but
when I check into 10.24.1.12, there is no this region.
And then, I stop all HBase cluster and start it. Regions locations are
re-structured and seems everything is OK.

In the log file of 10.24.1.12, I found following exceptions:

836118938_60020/hlog.dat.1236849158178, entries=100010. New log writer:
/hbase/log_10.24.1.12_1236836118938_60020/hlog.dat.1236849168393
2009-03-12 17:12:49,298 INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region
TESTTABLE,13575565132@2008-12-0117:16:55.117,1236847258901 in 48sec
2009-03-12 17:12:49,298 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Starting split of region
TESTTABLE,13575565132@2008-12-0117:16:55.117,1236847258901
2009-03-12 17:12:50,648 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed TESTTABLE,13575565132@2008-12-01 17:16:55.117,1236847258901
2009-03-12 17:12:50,809 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region TESTTABLE,13575565132@2008-12-0117:16:55.117,1236849169299/1762744366
available
2009-03-12 17:12:50,809 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed TESTTABLE,13575565132@2008-12-01 17:16:55.117,1236849169299
2009-03-12 17:12:50,865 INFO org.apache.hadoop.hbase.regionserver.HRegion:
region TESTTABLE,13575590622@2008-12-1615:49:40.143,1236849169299/1344805089
available
2009-03-12 17:12:50,865 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed TESTTABLE,13575590622@2008-12-16 15:49:40.143,1236849169299
2009-03-12 17:29:15,495 WARN org.apache.hadoop.hbase.RegionHistorian: Unable
to 'Region split from: WAPCDR,13575565132@2008-12-0117:16:55.117,1236847258901'
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server Some server for region , row
'TESTTABLE,13575565132@2008-12-0117:16:55.117,1236849169299', but
failed after 11 attempts.
Exceptions:
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1546)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)

org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2065)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1546)
        at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:632)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:895)

org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message