hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: HBase performance degradation (0.90.3)
Date Thu, 07 Jul 2011 18:05:48 GMT
On Thu, Jul 7, 2011 at 10:58 AM, Vaibhav Puranik <vpuranik@gmail.com> wrote:
> We used to flush all of our tables every night. I disabled major
> compactions, removed the cron that flushed the tables and replaced it with a
> cron that does major compactions every night.
> Since then we started getting the following error in our logs:
> *org.apache.hadoop.hbase.regionserver.wal.HLog:* *Too* *many* *hlogs:*
> *logs=33*, *maxlogs=32*; *forcing* *flush* *of* *59* *regions*(*s*)

You sure this was not always in your logs or at least some form of it?

Maybe you were not seeing theses because you were flushing each night.
 The forced flush would make it so we could clear up wal logs.

The above message happens because we are trying to keep an upper bound
on how many outstanding wal logs a regionserver has outstanding.  We
don't want the case where there are hundreds, then the regionserver
fails, and we have hundreds to split.  So, we'll try and keep them
under an upper bound by flushing out the regions with the old edits.

> Whenever this happens, the regionserver performance degrades considerably.
> gets slow down a lot, writes are not affected. (rpc.metrics.get_avg_time)
> goes up. This problem lasts for about 30 to 40 minutes and after that the
> cluster recovers.

This is during the flush period?  Perhaps reinstitute your nightly flushing?

> Besides this we are also seeing that the minor compactions have stopped
> (after disabling major compaction).

This seems odd.  You are running with DEBUG?  And no regions are
compacting anymore.

> Do I need to make sure that I am flushing before I do major compaction at
> night? Do I need to cron minor compaction (HBaseAdmin.compact)?

None of the above.


View raw message