hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Compactions after bulk load
Date Fri, 20 Jul 2018 18:59:13 GMT
Have you checked the output from bulk load and see if there were lines in
the following form (from LoadIncrementalHFiles#splitStoreFile) ?

    LOG.info("HFile at " + hfilePath + " no longer fits inside a
single " + "region.
Splitting...");

In the server log, you should see log in the following form:

      if (LOG.isDebugEnabled()) {
        LOG.debug("Compacting " + file +
          ", keycount=" + keyCount +
          ", bloomtype=" + r.getBloomFilterType().toString() +
          ", size=" + TraditionalBinaryPrefix.long2String(r.length(), "",
1) +
          ", encoding=" + r.getHFileReader().getDataBlockEncoding() +
          ", seqNum=" + seqNum +
          (allFiles ? ", earliestPutTs=" + earliestPutTs: ""));
      }

where allFiles being true indicates major compaction.

The above should give you some idea of the cause for the compaction
activity.

Thanks

On Tue, Jul 17, 2018 at 11:12 AM Austin Heyne <aheyne@ccri.com> wrote:

> Hi all,
>
> I'm trying to bulk load a large amount of data into HBase. The bulk load
> succeeds but then HBase starts running compactions. My input files are
> typically ~5-6GB and there are over 3k files. I've used the same table
> splits for the bulk ingest and the bulk load so there should be no
> reason for hbase to run any compactions. However, I'm seeing it first
> start compacting the hfiles into 25+GB files and then into 200+GB files
> but didn't let it run any longer. Additionally, I've talked with another
> coworker who's tried this process in the past and he's experience the
> same thing, eventually giving up on the feature. My attempts have been
> on HBase 1.4.2. Does anyone have information on why HBase is insisting
> on running these compactions or how I can stop them? They are
> essentially breaking the feature for us.
>
> Thanks,
>
> --
> Austin L. Heyne
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message