hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Compactions no longer limited
Date Thu, 03 Sep 2009 01:05:17 GMT
Hi Ken,

Compactions are serviced by a thread which sleeps for a configurable interval and then wakes
to do work. As compaction requests are raised, they are queued and the thread is signaled
and wakes early. When a region server first starts up, a limit is imposed on how many compaction
requests can be serviced from the queue before the thread goes back to sleep. The limit is
gradually relaxed. The net effect is to draw out compactions over an initial "safe mode".
The reason this is done is to limit load on DFS when the cluster is starting. Compactions
during this period often process flushes from the last shutdown and can well trigger splits.
Around the 0.19 timeframe I had a 25 node test cluster pushing 1000 regions per node. Startup
compaction/split activity would crush DFS and prevent successful (re)starts. The mechanism
described here solved that issue.

    - Andy





________________________________
From: Ken Weiner <ken@gumgum.com>
To: hbase-user@hadoop.apache.org
Sent: Wednesday, September 2, 2009 5:50:53 PM
Subject: Compactions no longer limited

What does it mean when the following INFO message appears in the
RegionServer logs?  I generally see that message show up on all
RegionServers at the same time.

org.apache.hadoop.hbase.regionserver.HRegionServer: compactions no longer
limited

Thanks, Ken



      
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message