hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Gray <jg...@facebook.com>
Subject RE: Why does the default hbase.hstore.compactionThreshold is 3?
Date Wed, 07 Apr 2010 06:06:47 GMT
Shen,

You are right.  Currently the default flush size is 64MB, the compactionThreshold is 3, and
the splitSize/max.filesize is 256MB.  So we end up compacting into a 192MB file when filling
an empty region.

Take a look at HBASE-2375 (https://issues.apache.org/jira/browse/HBASE-2375).  That issue
deals with changing some of this behavior as well as the defaults.

JG

> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-
> Daniel Cryans
> Sent: Tuesday, April 06, 2010 9:18 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: Why does the default hbase.hstore.compactionThreshold is
> 3?
> 
> It does incremental compacting since you don't want to spend too much
> time doing the compactions, and you don't want to compact very large
> store files with much smaller ones (that would result in rewriting the
> same data x times per day). Looking at Store.compact, you can see this
> comment:
> 
>         // Here we select files for incremental compaction.
>         // The rule is: if the largest(oldest) one is more than twice
> the
>         // size of the second, skip the largest, and continue to
> next...,
>         // until we meet the compactionThreshold limit.
> 
> So you normally have 1 big store file and a bunch of smaller ones on a
> production system (the major compaction creates the very big one). You
> won't end up with 1 after each minor compaction, or at least it's not
> very probable.
> 
> hbase.hregion.max.filesize is for the max size of a single store file
> in a single store (family), so it takes a few compactions to get a
> split.
> 
> Hope this helps,
> 
> J-D
> 
> On Tue, Apr 6, 2010 at 7:21 PM, ChingShen <chingshenchen@gmail.com>
> wrote:
> > Hi,
> >
> >  I got when the menstore reaches a configurable size(64MB), it's
> flushed to
> > HDFS, and create a new StoreFile, therefore,  when these StoreFiles
> more
> > than 3 files, they will be compacted to a single StoreFile. But, if
> the
> > default hbase.hstore.compactionThreshold is 3, does it mean that a
> compacted
> > StoreFile size is 192MB? or larger?  and what's relationship between
> > hbase.hstore.compactionThreshold and hbase.hregion.max.filesize?
> >
> > Thanks.
> >
> > Shen
> >

Mime
View raw message