hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <sean.bus...@gmail.com>
Subject Re: Mob reference tags go missing
Date Sat, 17 Aug 2019 15:05:20 GMT
Also the hfile version setting and hbase version on your masters, presuming
you are using the mob compaction chore in the master.

On Sat, Aug 17, 2019, 09:59 Sean Busbey <sean.busbey@gmail.com> wrote:

> That sounds like some of your hfiles are being written as hfile v2 instead
> of v3. Can you check versions and configs on all your region servers?
>
> On Sat, Aug 17, 2019, 08:42 Omri Cohen <Omri_cohen@outlook.com> wrote:
>
>> Sorry, accidentally pressed "send" before i finished the massage.
>>
>> Hello,
>>
>> We recently encountered a problem in our production hbase cluster (CDH
>> deployment, version 5.13.1).
>>
>> We have a cluster with a high mob percentage ( > 50% of objects are MOB,
>> the threshold is the default 102400 bytes). The cluster has been active for
>> about a year. Recently, our clients started to receive the mob reference
>> instead of the mob data when trying to access certain rows (in most tables
>> about 10% of the rows are bad, in one table it is about 50% of rows).
>>
>> When we investigated the HFiles, we saw that the "bad" cells are missing
>> two tags that exist in the "good" rows, it looks something like that:
>>
>> K: {rowkey of a "good" row} {Column family, column qualifier}...
>> vlen=76/seqid=... V: \x00\x04\xFB.{MOB file name} T[0]:  T[1]: {table name}
>> K: {rowkey of a "bad" row} {Column family, column qualifier}...
>> vlen=76/seqid=... V: \x00\x13\x1A{MOB file name}
>>
>> The "good" row returns the expected data when queried, while the "bad"
>> row returned the mob file reference instead of the data. This happened when
>> we queried from the REST server, the native java client, and the hbase
>> shell.
>>
>> When we examined the mob files, we saw that all the data was there.
>>
>>   *   Has anyone encountered a similar situation?
>>   *   Is it possible to manually add the missing mob reference tags to
>> the "bad" rows?
>>
>> No changes where made to the table in recently. We never encountered this
>> issue before.
>>
>> We would appreciate any help on this issue.
>> ________________________________
>> From: Omri Cohen
>> Sent: Saturday, August 17, 2019 4:28 PM
>> To: hbase-user@hadoop.apache.org <hbase-user@hadoop.apache.org>
>> Subject: Mob reference tags go missing
>>
>> Hello,
>>
>> We recently encountered a problem in our production hbase cluster.
>> We have a cluster with a high mob percentage ( > 50% of objects are MOB,
>> the threshold is the default 102400 bytes). The cluster has been active for
>> about a year. Recently, our clients started to receive the mob reference
>> instead of the mob data when trying to access certain rows (in most tables
>> about 10% of the rows are bad, in one table it is about 50% of rows).
>>
>> When we investigated the HFiles, we saw that the "bad" cells are missing
>> two tags that exist in the "good" rows, it looks something like that:
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message