hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin GEORGES <ke...@d33d33.fr>
Subject Re: Asked to modify this region's memstoreSize to a negative value which is incorrect
Date Tue, 15 May 2018 17:55:11 GMT
Yes, we load org.apache.hadoop.hbase.coprocessor.example.BulkDeleteEndpoint

No idea about how we led to this situation :/
We have had some trouble with some RS last week, which lead us to perform multiple hbase hack
-repair - not sure it’s related


On May 15, 2018 at 7:35:11 PM, rahul gidwani (rahul.gidwani@gmail.com) wrote:

Are you using coprocessors? Can you tell us any more about what led to  
this.  

Thanks  


On Tue, May 15, 2018 at 4:27 AM Kevin GEORGES <kevin@d33d33.fr> wrote:  

> We are running HBASE 1.4.0  
>  
>  
> On May 15, 2018 at 1:10:15 PM, Kevin GEORGES (kevin@d33d33.fr) wrote:  
>  
> Hello,  
>  
> We find region server abort with the following exception:  
>  
> 2018-05-15 08:23:23,920 ERROR  
> [RpcServer.default.FPBQ.Fifo.handler=27,queue=7,port=16020]  
> regionserver.HRegion: Asked to modify this region's  
> (continuum,R\x0C\xF6\xF2\xBD\xD4L"\xB5\xFC\  
> xC6b\x8D\xD7\xC8x$\x7F\xFA\x9F\xA4\x92,1524491062878.117704beb050dcd3920335b4b290a898.)
 
> memstoreSize to a negative value which is incorrect. Current  
> memstoreSize=-1533230, delta=480  
> java.lang.Exception  
> at  
> org.apache.hadoop.hbase.regionserver.HRegion.addAndGetGlobalMemstoreSize(HRegion.java:1205)
 
> at  
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3534) 

> at  
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3102)  
> at  
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3044)  
> at  
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:894)
 
> at  
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:822)
 
> at  
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2376)  
> at  
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36621)
 
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2352)  
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)  
> at  
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)  
> at  
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)  
> 2018-05-15 08:23:23,922 FATAL [regionserver/  
> dn-35.hadoop.B.GRA.infra.metrics.ovh.net/10.0.0.35:16020-splits-1525859600420]  
> regionserver.HRegionServer: ABORTING region server dn-35.hadoo  
> p.b.gra.infra.metrics.ovh.net,16020,1525859460041: Assertion failed while  
> closing store  
> continuum,RH\xFD\xA6\x88\xD7\xFB5\xBBq\xD9\xE8|\xF2I_r\x7F\xFA\xB0Y\x88,1509095154547.51aea042f53
 
> 655350c0d098fd378ab9b. v. flushableSize expected=0, actual= 23429. Current  
> memstoreSize=-34925. Maybe a coprocessor operation failed and left the  
> memstore in a partially updated state.  
> 2018-05-15 08:23:23,922 FATAL [regionserver/  
> dn-35.hadoop.B.GRA.infra.metrics.ovh.net/10.0.0.35:16020-splits-1525859600420]  
> regionserver.HRegionServer: RegionServer abort: loaded coproce  
> ssors are: [org.apache.hadoop.hbase.coprocessor.example.BulkDeleteEndpoint  
>  
>  
> The error about memstoreSize becoming negative appear at a steady rate  
> before abort (hundreds/sec)  
>  
> Any ideas?  
>  
> Thanks,  
>  
> Kevin  

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message