hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Billy Pearson" <sa...@pearsonwholesale.com>
Subject Re: how many rows per GB memory
Date Mon, 27 Oct 2008 19:29:51 GMT
I understand the memory requirements needed to hold all the memcache for 
writes.

Say I have 8 regions each with 4 families how much memory should it take to 
load the index files and do read request on the regions assuming no write 
will happen.

Currently I have compression turned on for the family and they hold around 
900M rows total but use 1.3GB of
memory when the region server is started and done loading regions for the 
table.
Is there a way to lower that memory usage?

If I add up all the index files that are with the data mapfiles they total 
only 173MB
So do we hold only index data for reads if so why is my region server using 
1.3GB vs something like 173MB on start up?

Billy




"Jim Kellerman (POWERSET)" 
<Jim.Kellerman@microsoft.com> wrote in message 
news:EAFDEC03CDA5D644878904F4A8F0158A5FE9070954@NA-EXMSG-C103.redmond.corp.microsoft.com...
The 64MB cache flush size is configurable.

---
Jim Kellerman, Powerset (Live Search, Microsoft Corporation)


> -----Original Message-----
> From: news [mailto:news@ger.gmane.org] On Behalf 
> Of Billy Pearson
> Sent: Monday, October 27, 2008 12:07 PM
> To: hbase-user@hadoop.apache.org
> Subject: Re: how many rows per GB memory
>
> Is the 64MB changeable or is that a hard limit in the code?
> Billy
>
> "Jim Kellerman (POWERSET)"
> <Jim.Kellerman@microsoft.com> wrote in message
> news:EAFDEC03CDA5D644878904F4A8F0158A5FE9070743@NA-EXMSG-
> C103.redmond.corp.microsoft.com...
> Each memcache will consume approximately
>  ((x bytes per key (HStoreKey = row + column + timestamp)) +
>   (avg value size))
>  * (number of rows)
> which is limited to 64MB per memcache.
>
> The heap size required is determined by:
>  (number of regions being hosted) * (number of families) * 64MB
>
> ---
> Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
>
>
> > -----Original Message-----
> > From: news [mailto:news@ger.gmane.org] On 
> > Behalf
> > Of Billy Pearson
> > Sent: Sunday, October 26, 2008 11:23 PM
> > To: hbase-user@hadoop.apache.org
> > Subject: how many rows per GB memory
> >
> > Is there some numbers I can figure on per GB of heap that I can get in
> to
> > a
> > regionserver?
> >
> > say something like this:
> > (x bytes avg per rowkey * max rows) / index interval = y GB Heap
> >
> >
> > Billy
> >
> >
> >
> >
>
>
>




Mime
View raw message