hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Beaudreault <bbeaudrea...@hubspot.com>
Subject Re: state-of-the-art method for merging regions on v0.94
Date Thu, 28 Aug 2014 22:58:41 GMT
Lars, so it worked for me, and I'm more than happy for anyone to use/adapt
it as necessary for hbase proper.  But I'm not sure it's anywhere near
production ready, and I don't have the time to work on it more right now.
 Perhaps someone with more knowledge of region internals could vet it and
add relevant tests.  We could enter a JIRA, and if I find time in the
future I can take a look.

And yes, @JM, my gist was specific to an online migration (cluster is
active, but table is disabled).  Offline did not meet our requirements at
the time, so I never tried it.



On Thu, Aug 28, 2014 at 5:04 PM, lars hofhansl <larsh@apache.org> wrote:

> Agreed.
>
> Bryan, we should pull in your code if that works better.
>
> -- Lars
>
>
>
> ________________________________
>  From: Andrew Purtell <apurtell@apache.org>
> To: "user@hbase.apache.org" <user@hbase.apache.org>
> Cc: Development <Development@mentacapital.com>
> Sent: Thursday, August 28, 2014 12:12 PM
> Subject: Re: state-of-the-art method for merging regions on v0.94
>
>
> If the 0.94 merge code doesn't work out the box we should fix that.
>
>
>
>
>
> On Thu, Aug 28, 2014 at 11:26 AM, Bryan Beaudreault <
> bbeaudreault@hubspot.com> wrote:
>
> > I've done it.  This is the code I used:
> > https://gist.github.com/bbeaudreault/7567385
> >
> > It comes from the hbase source, but is modified to actually work (the
> class
> > provided in hbase is private and does not work out of the box). There is
> a
> > readme at the bottom of the gist with my process.  One important note
> > though, I did this with a deep understanding (after hours of reading
> hbase
> > code and doing tests on a test cluster) of how it all works.  And even
> then
> > I felt nervous to do it in prod.  Hence why I went the snapshot/compact
> > route.
> >
> > I would definitely test it on a test cluster and get some familiarity
> > before getting close to a production table.  That said, I've run this on
> > 8-10 production tables a few months ago, reducing in size from 10-20x in
> > some cases.
> >
> >
> > On Thu, Aug 28, 2014 at 2:19 PM, Ted Tuttle <ted@mentacapital.com>
> wrote:
> >
> > > Hello-
> > >
> > > We recently realized our region size is 1G and need to increase it to
> get
> > > our region count under control.  I've done some research on merging
> > regions
> > > and have come away confused.
> > >
> > > There is the ops handbook:
> > >
> > > http://hbase.apache.org/book/ops.regionmgt.html
> > >
> > > And then there is this horror story:
> > >
> > > http://metabroadcast.com/blog/so-you-broke-hbase
> > >
> > > Is there someone out there that has done a large scale (i.e. 10:1
> > > reduction on 10k's of regions) merge successfully on HBase 0.94?  If
> so,
> > > how did you do it?
> > >
> > > Thanks,
> > > Ted
> > >
> > >
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message