hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pankaj kr <pankaj...@huawei.com>
Subject IndexOutOfBoundsException during retrieving region split point
Date Fri, 03 Jun 2016 15:12:44 GMT
Hi,

We met a weird scenario in our production environment.
IndexOutOfBoundsException is thrown while retrieving mid key of the storefile after region
compaction.

Log Snippet :
-----------------
2016-05-30 01:41:58,484 | INFO  | regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749
| Completed compaction of 1 (all) file(s) in CF of User_Namespace:User_Table,100050007010803_20140126_308010717550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3c5.
into eee1f433635d478197b212e2e378fce8(size=22.0 G), total size for store is 22.0 G. This selection
was in queue for 0sec, and took 6mins, 25sec to execute. | org.apache.hadoop.hbase.regionserver.HStore.logCompactionEndMessage(HStore.java:1356)
2016-05-30 01:41:58,485 | INFO  | regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749
| Completed compaction: Request = regionName=User_Namespace:User_Table,100050007010803_20140126_308010717550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3c5.,
storeName=CF, fileCount=1, fileSize=44.0 G, priority=6, time=295643974900644; duration=6mins,
25sec | org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:544)
2016-05-30 01:41:58,529 | ERROR | regionserver/RS-HOSTNAME/RS-IP:21302-longCompactions-1464247799749
| Compaction failed Request = regionName=User_Namespace:User_Table,100050007010803_20140126_308010717550001_756781_99_36_0_01,1464543296529.676ee7e9902c066b0e8c15745463d3c5.,
storeName=CF, fileCount=1, fileSize=44.0 G, priority=6, time=295643974900644 | org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:563)
java.lang.IndexOutOfBoundsException
                at java.nio.Buffer.checkIndex(Buffer.java:540)
                at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:139)
                at org.apache.hadoop.hbase.util.ByteBufferUtils.toBytes(ByteBufferUtils.java:490)
                at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.midkey(HFileBlockIndex.java:349)
                at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.midkey(HFileReaderV2.java:512)
                at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.midkey(StoreFile.java:1480)
                at org.apache.hadoop.hbase.regionserver.StoreFile.getFileSplitPoint(StoreFile.java:685)
                at org.apache.hadoop.hbase.regionserver.DefaultStoreFileManager.getSplitPoint(DefaultStoreFileManager.java:126)
                at org.apache.hadoop.hbase.regionserver.HStore.getSplitPoint(HStore.java:1986)
                at org.apache.hadoop.hbase.regionserver.RegionSplitPolicy.getSplitPoint(RegionSplitPolicy.java:82)
                at org.apache.hadoop.hbase.regionserver.HRegion.checkSplit(HRegion.java:7914)
                at org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestSplit(CompactSplitThread.java:240)
                at org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:552)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                at java.lang.Thread.run(Thread.java:745)

-----------------
Observation:
>> HFilePrettyPrinter also print the message "Unable to retrieve the midkey" for the
mid key.
>> HDFS fsck report the hfile healthy.

Though successful region compaction were also there, only few region compaction failed with
same error.

Have anyone faced this issue? Any help will be appreciated.
HBase version is 1.0.0.

Regards,
Pankaj

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message