hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <saint....@gmail.com>
Subject Re: Hbck errors
Date Sun, 06 Mar 2011 04:28:19 GMT
Are the daughter regions in the meta table?  Are both there?   Are they both online?

You could try enabling the parent region



On Mar 5, 2011, at 16:22, Marc Limotte <mslimotte@gmail.com> wrote:

> We had an issue a day ago with some OOME's on the region servers.  The
> master shutdown ok, but most of the RegionServers didn't and so eventually
> had to kill -9 them.  Brought it all back up and ran a major compaction to
> change the hbase block size.  This seemed to work, but now we have an
> inconsistency which is preventing bulk loads from continuing.
> 
> hbase hbck -details finds an inconsistency.  I tried -fix, but no help.
> *Chain of regions in table opx_ad_event_v2 is broken; edges does not contain
> advertiser^BOpenX PSA^Acountry^BSerbia^Apublisher^BDSNR - Filesharing
> ROW^Adomain^Bthejesperbay.com^Aadvertiser_tag^Bmmx.travel^Astarttime^B1295175600
> *
> * *
> hbck also notes that this region is offline:
> 
> *11/03/05 23:33:16 DEBUG util.HBaseFsck: Region
> opx_ad_event_v2,advertiser\x02OpenX
> PSA\x01country\x02Serbia\x01publisher\x02DSNR - Filesharing ROW\x01domain\
> x02thejesperbay.com\x01advertiser_tag\x02mmx.travel\x01starttime\x021295175600,1297185243218.6147d3696ba9db3a85e3afd08d0bc59a.
> offline, split, parent, ignoring.*
> 
> Looking in .META. I see that the region is indeed offline, and appears to be
> split:
> 
> info:regioninfo                           timestamp=1299301154675 ...
> OFFLINE => true,
> info:splitA                                  timestamp=1299283401019
> info:splitB                                  timestamp=1299283401019
> (full .META. row below)
> 
> So, I'm guessing that it was in the midst of splitting and did not complete.
> 
> How can I recover from this situation?
> 
> thanks,
> Marc
> 
> ----------- .META. output ----------------
> 
> hbase(main):001:0> get '.META.' , "opx_ad_event_v2,advertiser\x02OpenX
> PSA\x01country\x02Serbia\x01publisher\x02DSNR - Filesharing ROW\x01domain\
> x02thejesperbay.com\x01advert""
> COLUMN
> CELL
> 
> info:regioninfo                              timestamp=1299301154675,
> value=REGION => {NAME => 'opx_ad_event_v2,advertiser\x02OpenX
> PSA\x01country\x02Serbia\x01publisher\x02DS
>                                              NR - Filesharing
> ROW\x01domain\x02thejesperbay.com\x01advertiser_tag\x02mmx.travel
> \x01starttime\x021295175600,1297185243218.6147d3
>                                              696ba9db3a85e3afd08d0bc59a.',
> STARTKEY => 'advertiser\x02OpenX
> PSA\x01country\x02Serbia\x01publisher\x02DSNR - Filesharing ROW\x01
>                                              domain\x02thejesperbay.com
> \x01advertiser_tag\x02mmx.travel\x01starttime\x021295175600', ENDKEY =>
> 'advertiser\x02OpenX PSA\x01coun
>                                              try\x02United Arab
> Emirates\x01publisher\
> x02www.sixbillionsecrets.com/\x01advertiser_tag\x02mmx.arts and
> entertainment\x01publishe
>                                              r_tag\x02mmx.arts and
> entertainment\x01starttime\x021295877600', ENCODED =>
> 6147d3696ba9db3a85e3afd08d0bc59a, OFFLINE => true, SPL
>                                              IT => true, TABLE => {{NAME =>
> 'opx_ad_event_v2', FAMILIES => [{NAME => 'metrics', BLOOMFILTER => 'NONE',
> REPLICATION_SCOPE => '0'
>                                              , VERSIONS => '1', COMPRESSION
> => 'GZ', TTL => '2147483647', BLOCKSIZE => '1048576', IN_MEMORY => 'false',
> BLOCKCACHE => 'true'},
>                                              {NAME => 'topn', BLOOMFILTER
> => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'GZ', VERSIONS => '1',
> TTL => '2147483647', BLOCK
>                                              SIZE => '1048576', IN_MEMORY
> => 'false', BLOCKCACHE =>
> 'true'}]}}
> info:server                                  timestamp=1299181144063,
> value=ip-10-17-24-121.ec2.internal:60020
> 
> info:serverstartcode                         timestamp=1299181144063,
> value=1299180905510
> 
> info:splitA                                  timestamp=1299283401019,
> value=REGION => {NAME => 'opx_ad_event_v2,advertiser\x02OpenX
> PSA\x01country\x02Serbia\x01publisher\x02DS
>                                              NR - Filesharing
> ROW\x01domain\x02thejesperbay.com\x01advertiser_tag\x02mmx.travel
> \x01starttime\x021295175600,1299283399612.3b278f
>                                              1b0ea78af239409efc4f0b2a3d.',
> STARTKEY => 'advertiser\x02OpenX
> PSA\x01country\x02Serbia\x01publisher\x02DSNR - Filesharing ROW\x01
>                                              domain\x02thejesperbay.com
> \x01advertiser_tag\x02mmx.travel\x01starttime\x021295175600', ENDKEY =>
> 'advertiser\x02OpenX PSA\x01coun
>                                              try\x02Taiwan\x01domain\
> x02kanzhongguo.com\x01advertiser_tag\x02mmx.arts and
> entertainment\x01publisher_tag\x02\x01starttime\x0212
>                                              96910800', ENCODED =>
> 3b278f1b0ea78af239409efc4f0b2a3d, TABLE => {{NAME => 'opx_ad_event_v2',
> FAMILIES => [{NAME => 'metrics', BLO
>                                              OMFILTER => 'NONE',
> REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'GZ', TTL =>
> '2147483647', BLOCKSIZE => '65536', IN_
>                                              MEMORY => 'false', BLOCKCACHE
> => 'true'}, {NAME => 'topn', BLOOMFILTER => 'NONE', REPLICATION_SCOPE =>
> '0', VERSIONS => '1', COMPR
>                                              ESSION => 'GZ', TTL =>
> '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> 'true'}]}}
> info:splitB                                  timestamp=1299283401019,
> value=REGION => {NAME => 'opx_ad_event_v2,advertiser\x02OpenX
> PSA\x01country\x02Taiwan\x01domain\x02kanzh
> 
> ongguo.com\x01advertiser_tag\x02mmx.arts
> and
> entertainment\x01publisher_tag\x02\x01starttime\x021296910800,1299283399612.9d4164620
>                                              2a363812d068792311d3a9b.',
> STARTKEY => 'advertiser\x02OpenX PSA\x01country\x02Taiwan\x01domain\
> x02kanzhongguo.com\x01advertiser_ta
>                                              g\x02mmx.arts and
> entertainment\x01publisher_tag\x02\x01starttime\x021296910800', ENDKEY =>
> 'advertiser\x02OpenX PSA\x01country\x0
>                                              2United Arab
> Emirates\x01publisher\
> x02www.sixbillionsecrets.com/\x01advertiser_tag\x02mmx.arts and
> entertainment\x01publisher_tag\
>                                              x02mmx.arts and
> entertainment\x01starttime\x021295877600', ENCODED =>
> 9d41646202a363812d068792311d3a9b, TABLE => {{NAME => 'opx_ad
>                                              _event_v2', FAMILIES => [{NAME
> => 'metrics', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS =>
> '1', COMPRESSION => 'GZ'
>                                              , TTL => '2147483647',
> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME =>
> 'topn', BLOOMFILTER => 'NONE',
>                                              REPLICATION_SCOPE => '0',
> VERSIONS => '1', COMPRESSION => 'GZ', TTL => '2147483647', BLOCKSIZE =>
> '65536', IN_MEMORY => 'false', B
>                                              LOCKCACHE =>
> 'true'}]}}
> 
> 5 row(s) in 0.3960 seconds
> 
> 
> 
> ------------ hbck -details output -------------
> ...
> ERROR: Region
> hdfs://ip-10-17-5-253.ec2.internal:9000/hbase/opx_ad_event_v2/34a0ffe60da97431a809f0ffe8e5328a
> on HDFS, but not listed in META or deployed on any region server.
> ERROR: Region
> hdfs://ip-10-17-5-253.ec2.internal:9000/hbase/opx_ad_event_v2/3b278f1b0ea78af239409efc4f0b2a3d
> on HDFS, but not listed in META or deployed on any region server.
> *11/03/05 23:33:16 DEBUG util.HBaseFsck: Region
> opx_ad_event_v2,advertiser\x02OpenX
> PSA\x01country\x02Serbia\x01publisher\x02DSNR - Filesharing ROW\x01domain\
> x02thejesperbay.com\x01advertiser_tag\x02mmx.travel\x01starttime\x021295175600,1297185243218.6147d3696ba9db3a85e3afd08d0bc59a.
> offline, split, parent, ignoring.
> *ERROR: Region
> hdfs://ip-10-17-5-253.ec2.internal:9000/hbase/opx_ad_event_v2/9d41646202a363812d068792311d3a9b
> on HDFS, but not listed in META or deployed on any region server.
> *Chain of regions in table opx_ad_event_v2 is broken; edges does not contain
> advertiser^BOpenX PSA^Acountry^BSerbia^Apublisher^BDSNR - Filesharing
> ROW^Adomain^Bthejesperbay.com^Aadvertiser_tag^Bmmx.travel^Astarttime^B1295175600
> E*RROR: Found inconsistency in table opx_ad_event_v2
> Summary:
>  -ROOT- is okay.
>    Number of regions: 1
>    Deployed on:  ip-10-17-24-121.ec2.internal:60020
>  .META. is okay.
>    Number of regions: 1
>    Deployed on:  ip-10-17-5-252.ec2.internal:60020
> ...
> Chain of regions in table opx_ad_event_v2 is broken; edges does not contain
> advertiser^BOpenX PSA^Acountry^BSerbia^Apublisher^BDSNR - Filesharing
> ROW^Adomain^Bthejesperbay.com^Aadvertiser_tag^Bmmx.travel^Astarttime^B1295175600
> Table opx_ad_event_v2 is inconsistent.
>    Number of regions: 1612
>    Deployed on: ...
> 4 inconsistencies detected.
> Status: INCONSISTENT

Mime
View raw message