hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Major compaction time
Date Wed, 09 Mar 2011 18:25:48 GMT
You are running w/ the defaults where major compaction runs every day?
 It depends on how your data is changing but at the extreme, a major
compaction is a rewrite of the total data set.

We'll usually suggest that folks turn off the auto-major compaction
and instead run it manually at time of low loading -- evening or w/e
dependent on traffic -- or that they manage it in some way; perhaps
some tables need the major compaction less than others.

As to how long it takes, it depends.  At the moment the compactor is a
single thread per region so it could take a while though 18 hours
seems like a long time.   We won't major compact files that have
already been major compacted, so unless your write pattern adds files
across all regions, the second major compaction should run to
completion in a shorter time.


On Wed, Mar 9, 2011 at 9:01 AM, Iulia Zidaru <iulia.zidaru@1and1.ro> wrote:
>  Hi all,
> We have a cluster with 6 machines, 875 regions / machine and 2.75TB data.
> The system receives a constant amount of inserts and deletes. The compaction
> took about 18 h (we looked at **hbase.regionserver.compactionQueueSize
> metric in Ganglia).
> How long does a Major compaction take on your systems? How long it is
> supposed to take? Could we improve this time?
> Thank you,
> Iulia
> --
> Iulia Zidaru
> Java Developer
> 1&1 Internet AG - Bucharest/Romania - Web Components Romania
> 18 Mircea Eliade St
> Sect 1, Bucharest
> RO Bucharest, 012015
> iulia.zidaru@1and1.ro
> 0040 31 223 9153

View raw message