hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Austin Heyne <ahe...@ccri.com>
Subject Re: Compactions after bulk load
Date Mon, 23 Jul 2018 13:58:32 GMT
Thanks for the feedback, I've been slammed with other tasks but will get 
to this soon as we get other things stable.

-Austin


On 07/20/2018 02:59 PM, Ted Yu wrote:
> Have you checked the output from bulk load and see if there were lines in
> the following form (from LoadIncrementalHFiles#splitStoreFile) ?
>
>      LOG.info("HFile at " + hfilePath + " no longer fits inside a
> single " + "region.
> Splitting...");
>
> In the server log, you should see log in the following form:
>
>        if (LOG.isDebugEnabled()) {
>          LOG.debug("Compacting " + file +
>            ", keycount=" + keyCount +
>            ", bloomtype=" + r.getBloomFilterType().toString() +
>            ", size=" + TraditionalBinaryPrefix.long2String(r.length(), "",
> 1) +
>            ", encoding=" + r.getHFileReader().getDataBlockEncoding() +
>            ", seqNum=" + seqNum +
>            (allFiles ? ", earliestPutTs=" + earliestPutTs: ""));
>        }
>
> where allFiles being true indicates major compaction.
>
> The above should give you some idea of the cause for the compaction
> activity.
>
> Thanks
>
> On Tue, Jul 17, 2018 at 11:12 AM Austin Heyne <aheyne@ccri.com> wrote:
>
>> Hi all,
>>
>> I'm trying to bulk load a large amount of data into HBase. The bulk load
>> succeeds but then HBase starts running compactions. My input files are
>> typically ~5-6GB and there are over 3k files. I've used the same table
>> splits for the bulk ingest and the bulk load so there should be no
>> reason for hbase to run any compactions. However, I'm seeing it first
>> start compacting the hfiles into 25+GB files and then into 200+GB files
>> but didn't let it run any longer. Additionally, I've talked with another
>> coworker who's tried this process in the past and he's experience the
>> same thing, eventually giving up on the feature. My attempts have been
>> on HBase 1.4.2. Does anyone have information on why HBase is insisting
>> on running these compactions or how I can stop them? They are
>> essentially breaking the feature for us.
>>
>> Thanks,
>>
>> --
>> Austin L. Heyne
>>
>>

-- 
Austin L. Heyne


Mime
View raw message