hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Batyrshin <0x62...@gmail.com>
Subject Re: Region-server crash: Added a key not lexically larger than previous
Date Sat, 17 Aug 2019 17:08:38 GMT
Replayd at  https://issues.apache.org/jira/browse/HBASE-22862 <https://issues.apache.org/jira/browse/HBASE-22862>

> On 16 Aug 2019, at 20:16, Stack <stack@duboce.net> wrote:
> 
> Interesting. The Cells differ in sequence id. Would seem to imply race
> around the getting of sequenceid writing the WAL. The crashes are happening
> on regionserver replay of WAL files recovering crashed server? Or when is
> it happening? If the former, you might have to find the offending WAL and
> move it out of the line-of-action. Will need to replay the WAL subsequently
> but w/o the bad Cell.
> S
> 
> On Thu, Aug 15, 2019 at 8:57 AM Alexander Batyrshin <0x62ash@gmail.com>
> wrote:
> 
>> 
>> Hello all,
>> 
>> We observer error "Added a key not lexically larger than previous” that
>> cause most of our region-servers to crash in our cluster.
>> HBase-1.4.10
>> 
>> 2019-08-15 18:02:10,554 INFO  [MemStoreFlusher.0] regionserver.HRegion:
>> Flushing 1/1 column families, memstore=56.08 MB
>> 2019-08-15 18:02:10,727 WARN  [MemStoreFlusher.0] regionserver.HStore:
>> Failed flushing store file, retrying num=0
>> java.io.IOException: Added a key not lexically larger than previous.
>> Current cell =
>> \x0901820448218>wGavb'/d:elr/1565881054828/DeleteColumn/vlen=0/seqid=44456567,
>> lastCell =
>> \x0901820448218>wGavb'/d:elr/1565881054828/Put/vlen=1/seqid=44457770
>>        at org.apache.hadoop.hbase.io
>> .hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:204)
>>        at org.apache.hadoop.hbase.io
>> .hfile.HFileWriterV2.append(HFileWriterV2.java:279)
>>        at org.apache.hadoop.hbase.io
>> .hfile.HFileWriterV3.append(HFileWriterV3.java:87)
>>        at
>> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1127)
>>        at
>> org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:139)
>>        at
>> org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
>>        at
>> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:1003)
>>        at
>> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2523)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2622)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2352)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2314)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2200)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2125)
>>        at
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:512)
>>        at
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:482)
>>        at
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:76)
>>        at
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:264)
>>        at java.lang.Thread.run(Thread.java:748)
>> 
>> 2019-08-15 18:02:21,776 WARN  [MemStoreFlusher.0] regionserver.HStore:
>> Failed flushing store file, retrying num=9
>> java.io.IOException: Added a key not lexically larger than previous.
>> Current cell =
>> \x0901820448218>wGavb'/d:elr/1565881054828/DeleteColumn/vlen=0/seqid=44456567,
>> lastCell =
>> \x0901820448218>wGavb'/d:elr/1565881054828/Put/vlen=1/seqid=44457770
>>        at org.apache.hadoop.hbase.io
>> .hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:204)
>>        at org.apache.hadoop.hbase.io
>> .hfile.HFileWriterV2.append(HFileWriterV2.java:279)
>>        at org.apache.hadoop.hbase.io
>> .hfile.HFileWriterV3.append(HFileWriterV3.java:87)
>>        at
>> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1127)
>>        at
>> org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:139)
>>        at
>> org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
>>        at
>> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:1003)
>>        at
>> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2523)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2622)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2352)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2314)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2200)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2125)
>>        at
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:512)
>>        at
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:482)
>>        at
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:76)
>>        at
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:264)
>>        at java.lang.Thread.run(Thread.java:748)
>> 
>> 2019-08-15 18:02:21,777 FATAL [MemStoreFlusher.0]
>> regionserver.HRegionServer: ABORTING region server
>> prod006,60020,1565873610692: Replay of WAL required. Forcing server shutdown
>> org.apache.hadoop.hbase.DroppedSnapshotException: region:
>> TBL_TABLE_CODE,\x0904606203097821slG=sPD,1563070299676.5110b3395ca64a51cea99c6572a4c3d9.
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2675)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2352)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2314)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2200)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2125)
>>        at
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:512)
>>        at
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:482)
>>        at
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:76)
>>        at
>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:264)
>>        at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.io.IOException: Added a key not lexically larger than
>> previous. Current cell =
>> \x0901820448218>wGavb'/d:elr/1565881054828/DeleteColumn/vlen=0/seqid=44456567,
>> lastCell =
>> \x0901820448218>wGavb'/d:elr/1565881054828/Put/vlen=1/seqid=44457770
>>        at org.apache.hadoop.hbase.io
>> .hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:204)
>>        at org.apache.hadoop.hbase.io
>> .hfile.HFileWriterV2.append(HFileWriterV2.java:279)
>>        at org.apache.hadoop.hbase.io
>> .hfile.HFileWriterV3.append(HFileWriterV3.java:87)
>>        at
>> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1127)
>>        at
>> org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:139)
>>        at
>> org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
>>        at
>> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:1003)
>>        at
>> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2523)
>>        at
>> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2622)
>>        ... 9 more
>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message