hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Region server memory requirements
Date Sat, 20 Dec 2008 03:13:03 GMT
Based on this, leaving the Hadoop default (128) might be the
way to go. 

Later, maybe it would make sense to dynamically set the index
interval based on the distribution of cell sizes in the 
mapfile at some future time, according to some parameterized
formula that could be adjusted with config variable(s). This
could be done during compaction. Would make sense to also
consider the distribution of key lengths. Or there could be
other similar tricks implemented to keep index sizes down. 

In my opinion, for 0.20.0, MapFile should be brought local so
we can begin hacking it over time into a new file format. I
was thinking that designing and/or implementing a wholly new
format such as TFile would block I/O improvements for a long

  - Andy

> From: stack <stack@duboce.net>
> Subject: Re: Region server memory requirements
> To: hbase-user@hadoop.apache.org
> Date: Friday, December 19, 2008, 6:22 PM
> I was just thinking on this.  Testing, changing the index
> interval has the biggest direct effect on memory used. 
> Currently its 32 which seems to be fine for cells of 1K and
> greater.  With 32, you can load a decent amount of regions
> per regionserver.  Smaller cells seem common enough though. 
> Here interval should be the hadoop, as opposed to hbase,
> default of 128 or even 1024 (Its 'low' to help
> improve access times).  It seems that for the case where
> cells are < 10 bytes or so, interval should be 1k or even
> 4k.  Should we just be conservative in defaults since OOME
> is worse than slow access times.
> Backporting HBASE-900 might be a bit much for 0.18.2.   We
> should for sure backport the fix that makes it so you can
> set the index interval to 0.18 branch.
> St.Ack
> Andrew Purtell wrote:
> > Does it make sense to backport HBASE-900 to 0.18.2?
> > 
> >    - Andy


View raw message