hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Hbck errors
Date Sun, 06 Mar 2011 17:18:35 GMT
So, yeah Marc, what are the rows that follow the ones you post below?
Are they the info:splitA and info:splitB or something else?
Thanks,
St.Ack

On Sat, Mar 5, 2011 at 4:22 PM, Marc Limotte <mslimotte@gmail.com> wrote:
> We had an issue a day ago with some OOME's on the region servers.  The
> master shutdown ok, but most of the RegionServers didn't and so eventually
> had to kill -9 them.  Brought it all back up and ran a major compaction to
> change the hbase block size.  This seemed to work, but now we have an
> inconsistency which is preventing bulk loads from continuing.
>
> hbase hbck -details finds an inconsistency.  I tried -fix, but no help.
> *Chain of regions in table opx_ad_event_v2 is broken; edges does not contain
> advertiser^BOpenX PSA^Acountry^BSerbia^Apublisher^BDSNR - Filesharing
> ROW^Adomain^Bthejesperbay.com^Aadvertiser_tag^Bmmx.travel^Astarttime^B1295175600
> *
> * *
> hbck also notes that this region is offline:
>
> *11/03/05 23:33:16 DEBUG util.HBaseFsck: Region
> opx_ad_event_v2,advertiser\x02OpenX
> PSA\x01country\x02Serbia\x01publisher\x02DSNR - Filesharing ROW\x01domain\
> x02thejesperbay.com\x01advertiser_tag\x02mmx.travel\x01starttime\x021295175600,1297185243218.6147d3696ba9db3a85e3afd08d0bc59a.
> offline, split, parent, ignoring.*
>
> Looking in .META. I see that the region is indeed offline, and appears to be
> split:
>
> info:regioninfo                           timestamp=1299301154675 ...
> OFFLINE => true,
> info:splitA                                  timestamp=1299283401019
> info:splitB                                  timestamp=1299283401019
> (full .META. row below)
>
> So, I'm guessing that it was in the midst of splitting and did not complete.
>
> How can I recover from this situation?
>
> thanks,
> Marc
>
> ----------- .META. output ----------------
>
> hbase(main):001:0> get '.META.' , "opx_ad_event_v2,advertiser\x02OpenX
> PSA\x01country\x02Serbia\x01publisher\x02DSNR - Filesharing ROW\x01domain\
> x02thejesperbay.com\x01advert""
> COLUMN
> CELL
>
>  info:regioninfo                              timestamp=1299301154675,
> value=REGION => {NAME => 'opx_ad_event_v2,advertiser\x02OpenX
> PSA\x01country\x02Serbia\x01publisher\x02DS
>                                              NR - Filesharing
> ROW\x01domain\x02thejesperbay.com\x01advertiser_tag\x02mmx.travel
> \x01starttime\x021295175600,1297185243218.6147d3
>                                              696ba9db3a85e3afd08d0bc59a.',
> STARTKEY => 'advertiser\x02OpenX
> PSA\x01country\x02Serbia\x01publisher\x02DSNR - Filesharing ROW\x01
>                                              domain\x02thejesperbay.com
> \x01advertiser_tag\x02mmx.travel\x01starttime\x021295175600', ENDKEY =>
> 'advertiser\x02OpenX PSA\x01coun
>                                              try\x02United Arab
> Emirates\x01publisher\
> x02www.sixbillionsecrets.com/\x01advertiser_tag\x02mmx.arts and
> entertainment\x01publishe
>                                              r_tag\x02mmx.arts
and
> entertainment\x01starttime\x021295877600', ENCODED =>
> 6147d3696ba9db3a85e3afd08d0bc59a, OFFLINE => true, SPL
>                                              IT => true, TABLE
=> {{NAME =>
> 'opx_ad_event_v2', FAMILIES => [{NAME => 'metrics', BLOOMFILTER => 'NONE',
> REPLICATION_SCOPE => '0'
>                                              , VERSIONS =>
'1', COMPRESSION
> => 'GZ', TTL => '2147483647', BLOCKSIZE => '1048576', IN_MEMORY => 'false',
> BLOCKCACHE => 'true'},
>                                              {NAME => 'topn',
BLOOMFILTER
> => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'GZ', VERSIONS => '1',
> TTL => '2147483647', BLOCK
>                                              SIZE => '1048576',
IN_MEMORY
> => 'false', BLOCKCACHE =>
> 'true'}]}}
>  info:server                                  timestamp=1299181144063,
> value=ip-10-17-24-121.ec2.internal:60020
>
>  info:serverstartcode                         timestamp=1299181144063,
> value=1299180905510
>
>  info:splitA                                  timestamp=1299283401019,
> value=REGION => {NAME => 'opx_ad_event_v2,advertiser\x02OpenX
> PSA\x01country\x02Serbia\x01publisher\x02DS
>                                              NR - Filesharing
> ROW\x01domain\x02thejesperbay.com\x01advertiser_tag\x02mmx.travel
> \x01starttime\x021295175600,1299283399612.3b278f
>                                              1b0ea78af239409efc4f0b2a3d.',
> STARTKEY => 'advertiser\x02OpenX
> PSA\x01country\x02Serbia\x01publisher\x02DSNR - Filesharing ROW\x01
>                                              domain\x02thejesperbay.com
> \x01advertiser_tag\x02mmx.travel\x01starttime\x021295175600', ENDKEY =>
> 'advertiser\x02OpenX PSA\x01coun
>                                              try\x02Taiwan\x01domain\
> x02kanzhongguo.com\x01advertiser_tag\x02mmx.arts and
> entertainment\x01publisher_tag\x02\x01starttime\x0212
>                                              96910800', ENCODED
=>
> 3b278f1b0ea78af239409efc4f0b2a3d, TABLE => {{NAME => 'opx_ad_event_v2',
> FAMILIES => [{NAME => 'metrics', BLO
>                                              OMFILTER => 'NONE',
> REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'GZ', TTL =>
> '2147483647', BLOCKSIZE => '65536', IN_
>                                              MEMORY => 'false',
BLOCKCACHE
> => 'true'}, {NAME => 'topn', BLOOMFILTER => 'NONE', REPLICATION_SCOPE =>
> '0', VERSIONS => '1', COMPR
>                                              ESSION => 'GZ',
TTL =>
> '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> 'true'}]}}
>  info:splitB                                  timestamp=1299283401019,
> value=REGION => {NAME => 'opx_ad_event_v2,advertiser\x02OpenX
> PSA\x01country\x02Taiwan\x01domain\x02kanzh
>
> ongguo.com\x01advertiser_tag\x02mmx.arts
> and
> entertainment\x01publisher_tag\x02\x01starttime\x021296910800,1299283399612.9d4164620
>                                              2a363812d068792311d3a9b.',
> STARTKEY => 'advertiser\x02OpenX PSA\x01country\x02Taiwan\x01domain\
> x02kanzhongguo.com\x01advertiser_ta
>                                              g\x02mmx.arts and
> entertainment\x01publisher_tag\x02\x01starttime\x021296910800', ENDKEY =>
> 'advertiser\x02OpenX PSA\x01country\x0
>                                              2United Arab
> Emirates\x01publisher\
> x02www.sixbillionsecrets.com/\x01advertiser_tag\x02mmx.arts and
> entertainment\x01publisher_tag\
>                                              x02mmx.arts and
> entertainment\x01starttime\x021295877600', ENCODED =>
> 9d41646202a363812d068792311d3a9b, TABLE => {{NAME => 'opx_ad
>                                              _event_v2', FAMILIES
=> [{NAME
> => 'metrics', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS =>
> '1', COMPRESSION => 'GZ'
>                                              , TTL => '2147483647',
> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME =>
> 'topn', BLOOMFILTER => 'NONE',
>                                              REPLICATION_SCOPE
=> '0',
> VERSIONS => '1', COMPRESSION => 'GZ', TTL => '2147483647', BLOCKSIZE =>
> '65536', IN_MEMORY => 'false', B
>                                              LOCKCACHE =>
> 'true'}]}}
>
> 5 row(s) in 0.3960 seconds
>
>
>
> ------------ hbck -details output -------------
> ...
> ERROR: Region
> hdfs://ip-10-17-5-253.ec2.internal:9000/hbase/opx_ad_event_v2/34a0ffe60da97431a809f0ffe8e5328a
> on HDFS, but not listed in META or deployed on any region server.
> ERROR: Region
> hdfs://ip-10-17-5-253.ec2.internal:9000/hbase/opx_ad_event_v2/3b278f1b0ea78af239409efc4f0b2a3d
> on HDFS, but not listed in META or deployed on any region server.
> *11/03/05 23:33:16 DEBUG util.HBaseFsck: Region
> opx_ad_event_v2,advertiser\x02OpenX
> PSA\x01country\x02Serbia\x01publisher\x02DSNR - Filesharing ROW\x01domain\
> x02thejesperbay.com\x01advertiser_tag\x02mmx.travel\x01starttime\x021295175600,1297185243218.6147d3696ba9db3a85e3afd08d0bc59a.
> offline, split, parent, ignoring.
> *ERROR: Region
> hdfs://ip-10-17-5-253.ec2.internal:9000/hbase/opx_ad_event_v2/9d41646202a363812d068792311d3a9b
> on HDFS, but not listed in META or deployed on any region server.
> *Chain of regions in table opx_ad_event_v2 is broken; edges does not contain
> advertiser^BOpenX PSA^Acountry^BSerbia^Apublisher^BDSNR - Filesharing
> ROW^Adomain^Bthejesperbay.com^Aadvertiser_tag^Bmmx.travel^Astarttime^B1295175600
> E*RROR: Found inconsistency in table opx_ad_event_v2
> Summary:
>  -ROOT- is okay.
>    Number of regions: 1
>    Deployed on:  ip-10-17-24-121.ec2.internal:60020
>  .META. is okay.
>    Number of regions: 1
>    Deployed on:  ip-10-17-5-252.ec2.internal:60020
> ...
> Chain of regions in table opx_ad_event_v2 is broken; edges does not contain
> advertiser^BOpenX PSA^Acountry^BSerbia^Apublisher^BDSNR - Filesharing
> ROW^Adomain^Bthejesperbay.com^Aadvertiser_tag^Bmmx.travel^Astarttime^B1295175600
> Table opx_ad_event_v2 is inconsistent.
>    Number of regions: 1612
>    Deployed on: ...
> 4 inconsistencies detected.
> Status: INCONSISTENT
>

Mime
View raw message