hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zach Johnson <zjohn...@ccri.com>
Subject Region stuck in transition Splitting
Date Fri, 11 Dec 2020 14:36:53 GMT
version: HBase 1.4.8

We have a stuck RIT that wont clear. The parent was offlined, and 
daughter region were created and moved out of the parents .split dir. 
But the parent region is never removed and the RIT doesn't clear. We 
have tried to run `move '<parent_region_name>'`, `unassign 
'<parent_region_name>'`, and |sudo stop hbase-regionserver && sudo start 
hbase-regionserver |on the parent and daughter regionservers. My only 
other thought is to disable the table, delete the parent region and 
enable the table. Any advice on how to clear the RIT or insight on what 
is causing it would be appreciated. I have included some logs bellow 
that might be helpful.

Thanks,

Zach

Region server logs:

    2020-12-04 14:00:35,430 INFO
    [RpcServer.priority.FPBQ.Fifo.handler=17,queue=7,port=16020]
    regionserver.RSRpcServices: Close d0b06e343bc8ae49ef7e5b66089fcb2d,
    moving to null
    2020-12-04 14:00:35,431 INFO
    [StoreCloserThread-synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.-1]
    regionserver.HStore: Closed d
    2020-12-04 14:00:35,436 INFO [RS_CLOSE_REGION-ip-10-0-1-4:16020-0]
    regionserver.HRegion: Closed
    synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
    2020-12-04 14:00:35,438 INFO
    [RpcServer.priority.FPBQ.Fifo.handler=17,queue=7,port=16020]
    regionserver.RSRpcServices: Open
    synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
    2020-12-04 14:00:35,703 INFO
    [StoreOpener-d0b06e343bc8ae49ef7e5b66089fcb2d-1] hfile.CacheConfig:
    Created cacheConfig for d:
    blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@58ff4bf2,
    cacheDataOnRead=true, cacheDataOnWrite=false,
    cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
    cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
    2020-12-04 14:00:35,703 INFO
    [StoreOpener-d0b06e343bc8ae49ef7e5b66089fcb2d-1]
    compactions.CompactionConfiguration: size [1409286144, 5000000000,
    5000000000); files [6, 10); ratio 1.000000; off-peak ratio 5.000000;
    throttle point 28185722880; major period 0, major jitter 0.500000,
    min locality to compact 0.000000; tiered compaction: max_age
    9223372036854775807, incoming window min 6, compaction policy for
    tiered window
    org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
    single output for minor true, compaction window factory
    org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
    2020-12-04 14:00:45,076 INFO [RS_OPEN_REGION-ip-10-0-1-4:16020-2]
    regionserver.HRegion: Onlined d0b06e343bc8ae49ef7e5b66089fcb2d; next
    sequenceid=2009960
    2020-12-04 14:00:45,077 INFO
    [PostOpenDeployTasks:d0b06e343bc8ae49ef7e5b66089fcb2d]
    regionserver.HRegionServer: Post open deploy tasks for
    synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
    2020-12-04 14:16:13,156 INFO
    [RpcServer.priority.FPBQ.Fifo.handler=17,queue=7,port=16020]
    regionserver.RSRpcServices: Splitting
    synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
    2020-12-04 14:16:20,003 INFO
    [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
    regionserver.SplitTransaction: Starting split of region
    synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
    2020-12-04 14:16:20,457 INFO
    [StoreCloserThread-synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.-1]
    regionserver.HStore: Closed d
    2020-12-04 14:16:20,462 INFO
    [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
    regionserver.HRegion: Closed
    synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
    2020-12-04 14:16:20,462 INFO
    [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
    regionserver.SplitTransaction: Preparing to split 14 storefiles for
    region
    synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
    using 8 threads
    2020-12-04 14:16:22,228 INFO
    [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
    s3n2.S3NativeFileSystem2: rename
    s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/d0b06e343bc8ae49ef7e5b66089fcb2d/.splits/453d941ff97a10bcf28e6ffe22d1d0e2
    s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/453d941ff97a10bcf28e6ffe22d1d0e2
    2020-12-04 14:16:23,988 INFO
    [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
    s3n2.S3NativeFileSystem2: rename
    s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/d0b06e343bc8ae49ef7e5b66089fcb2d/.splits/9d210ddbe593c83c62ac2d02413b7d97
    s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/9d210ddbe593c83c62ac2d02413b7d97
    2020-12-04 14:16:26,391 INFO
    [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
    regionserver.SplitRequest: Running rollback/cleanup of failed split
    of
    synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.;
    Failed
    ip-10-0-1-4.ec2.internal,16020,1606786380554-daughterOpener=9d210ddbe593c83c62ac2d02413b7d97


echo "scan 'hbase:meta'"   grep file for region

      synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:regioninfo, timestamp=1607091385153, value={ENCODED => d0b06e343bc8ae49ef7e5b66089fcb2d,
NAME => 'synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.',
STARTKEY => '\x018000017536e25cf0\x00e13', ENDKEY => '\x02', OFFLINE => true, SPLIT
=> true}
      synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:seqnumDuringOpen, timestamp=1607090445077, value=\x00\x00\x00\x00\x00\x1E\xABh
      synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:server, timestamp=1607090445077, value=ip-10-0-1-4.ec2.internal:16020
      synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:serverstartcode, timestamp=1607090445077, value=1606786380554
      synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:sn, timestamp=1606786396087, value=ip-10-0-1-4.ec2.internal,16020,1606786380554
      synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:splitA, timestamp=1607091385153, value={ENCODED => 453d941ff97a10bcf28e6ffe22d1d0e2,
NAME => 'synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1607091380003.453d941ff97a10bcf28e6ffe22d1d0e2.',
STARTKEY => '\x018000017536e25cf0\x00e13', ENDKEY => '\x01800001753f422970\x0063'}
      synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:splitB, timestamp=1607091385153, value={ENCODED => 9d210ddbe593c83c62ac2d02413b7d97,
NAME => 'synthetic_PRM_attr_dtg_v8_00609,\x01800001753f422970\x0063,1607091380003.9d210ddbe593c83c62ac2d02413b7d97.',
STARTKEY => '\x01800001753f422970\x0063', ENDKEY => '\x02'}
      synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:state, timestamp=1607091380003, value=SPLITTING


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message