hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yossi Ittach" <yoss...@gmail.com>
Subject Re: HBase region server - utilization extremely unbalanced
Date Thu, 23 Oct 2008 13:28:54 GMT
J-D,

I have only one table (Huge) , with only 1 family - which means every region
has exactly 1 file . Does it mean that I can significantly decrease the size
of the MapFile indexes in memory?
Also , what do you think will be the impact of increasing the region size
(from 256 to 512, for example) , in this scenario?



Vale et me ama
Yossi


On Thu, Oct 23, 2008 at 2:35 PM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:

> Yossi,
>
> Yeah they will go up because each datanode keeps their MapFile indexes in
> memory and the regionservers keep a Memcache of max 64MB (configurable, see
> hbase-default.xml) for each region it owns.
>
> Rule of thumb? Well in hbase-default the maximum a single family can grow
> inside a single region is 256MB so you can estimate the number of regions
> you will have, but it also depends on the number of tables and families.
> For
> example, if you have a single table with 10 equally filled families, you
> should expect around 12 regions. Only one family? 120 regions rough.
>
> So, based on that number of regions, you can extrapolate the memory needed
> to host your system. Big nodes with 16GB mem will host way more regions
> then
> a EC2 small instance.
>
> J-D
>
> On Thu, Oct 23, 2008 at 8:22 AM, Yossi Ittach <yossale@gmail.com> wrote:
>
> > Thanks for the quick reply.
> >
> > I'm following the jvm Memory consumption (using "top") , and what bothers
> > me
> > is that it seems the percentages are just going up and up , and it makes
> me
> > kind of worried.
> >
> > I'm trying to load the system with 30GB of data (this is a benchmark) . I
> > estimate that my production environment will require at least 3 times
> that
> > size.  Is there a rule-of-thumb as to how many region servers I'll need?
> >
> >
> > Vale et me ama
> > Yossi
> >
> >
> > On Thu, Oct 23, 2008 at 2:14 PM, Jean-Daniel Cryans <jdcryans@apache.org
> > >wrote:
> >
> > > Yossi,
> > >
> > > The META region is usually heavily used and it's worst when you use the
> > web
> > > UI. Just for the lolz, go on the Master's page (the main page) and hit
> > > "refresh" a couple of times; you should see that number go high up.
> > >
> > > And on how to avoid it, well the only way to split that load would be
> to
> > > have the META region do a split but it will require a lot of data hence
> a
> > > lot of user regions which I doubt you have on 2 machines.
> > >
> > > J-D
> > >
> > > On Thu, Oct 23, 2008 at 8:08 AM, Yossi Ittach <yossale@gmail.com>
> wrote:
> > >
> > > > Hi
> > > >
> > > > Using Hbase with 2 Region servers on similar machines , I see that
> one
> > > > machine is serving almost 400 requests per second , while the other
> one
> > > is
> > > > serving 0-10 . This cause extreme overload on the first machine. Any
> > idea
> > > > what causes it , or how it can be avoided?
> > > >
> > > > Thanks!
> > > >
> > > > Vale et me ama
> > > > Yossi
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message