hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Why does the default hbase.hstore.compactionThreshold is 3?
Date Wed, 07 Apr 2010 04:17:58 GMT
It does incremental compacting since you don't want to spend too much
time doing the compactions, and you don't want to compact very large
store files with much smaller ones (that would result in rewriting the
same data x times per day). Looking at Store.compact, you can see this
comment:

        // Here we select files for incremental compaction.
        // The rule is: if the largest(oldest) one is more than twice the
        // size of the second, skip the largest, and continue to next...,
        // until we meet the compactionThreshold limit.

So you normally have 1 big store file and a bunch of smaller ones on a
production system (the major compaction creates the very big one). You
won't end up with 1 after each minor compaction, or at least it's not
very probable.

hbase.hregion.max.filesize is for the max size of a single store file
in a single store (family), so it takes a few compactions to get a
split.

Hope this helps,

J-D

On Tue, Apr 6, 2010 at 7:21 PM, ChingShen <chingshenchen@gmail.com> wrote:
> Hi,
>
>  I got when the menstore reaches a configurable size(64MB), it's flushed to
> HDFS, and create a new StoreFile, therefore,  when these StoreFiles more
> than 3 files, they will be compacted to a single StoreFile. But, if the
> default hbase.hstore.compactionThreshold is 3, does it mean that a compacted
> StoreFile size is 192MB? or larger?  and what's relationship between
> hbase.hstore.compactionThreshold and hbase.hregion.max.filesize?
>
> Thanks.
>
> Shen
>

Mime
View raw message