hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Brondino <ibrond...@fi.upm.es>
Subject Re: splits and merge
Date Thu, 16 Jul 2015 15:04:31 GMT
Thanks Ted. That should work for us.

Regards,

Ivan

2015-07-16 16:06 GMT+02:00 Ted Yu <yuzhihong@gmail.com>:

> Ivan:
> You can utilize the following method in HBaseAdmin to retrieve compaction
> status:
>
>   public CompactionState getCompactionState(final TableName tableName)
>
> This API is available in 0.98+
>
> FYI
>
> On Thu, Jul 16, 2015 at 6:58 AM, Ivan Brondino <ibrondino@fi.upm.es>
> wrote:
>
> > Hi Shahab,
> >
> > We issue the splits on the table and right away the major compact of the
> > table.
> > Then we connect to the HRegionServer info server at http://host:60030
> and
> > we check html output for the task to be completed using an HttpClient.
> > So we actually wait for major compact and not for split.
> > We've been using it since 0.89 but we believe there must be a bettter
> > approach than that.
> >
> > Thanks,
> >
> > IVan
> >
> >
> >
> > 2015-07-16 15:50 GMT+02:00 Shahab Yunus <shahab.yunus@gmail.com>:
> >
> > > Ivan,
> > >
> > > Is it possible that you poll for number of regions (e.g. in a loop)
> after
> > > invoking split or merge to confirm that the action has been performed?
> I
> > > know it is a crude way but maybe something can be done in these lines.
> Or
> > > are you already doing this when you said 'look for Region Info'?
> > >
> > > Regards,
> > > Shahab
> > >
> > > On Thu, Jul 16, 2015 at 9:46 AM, Ivan Brondino <ibrondino@fi.upm.es>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I can confirm that it was what Jean Marc was saying. Thanks JM!
> > > >
> > > > Regarding question #1, any help? I am rephrasing it in case was not
> > > clear.
> > > >
> > > > 1) Since both split and merge are async invocations on the client
> side,
> > > we
> > > > have a naîve
> > > > workaround to make it synchronous that is to look at the RS Info
> Server
> > > via
> > > > an HttpClient checking
> > > > for pending tasks to be finished. Once all tasks are finished in all
> RS
> > > we
> > > > issue the next split/merge for the same table. Is there a better way
> to
> > > > check if major compact has finished?
> > > >
> > > > Thanks again,
> > > >
> > > > Iván
> > > >
> > > >
> > > > 2015-07-15 17:49 GMT+02:00 Ivan Brondino <ibrondino@fi.upm.es>:
> > > >
> > > > > Hi Ted, Jean Marc,
> > > > >
> > > > > Thank you guys for your answer.
> > > > >
> > > > > I think Jean Marc spotted the issue, although we do slightly
> > different.
> > > > >
> > > > > Let's say we have regions A and B. We split both A  into A1 and A2
> > and
> > > B
> > > > > into B1 and B2.
> > > > > Then we merge A2 and B1. Note that A and B are adjacent and so A1
> > with
> > > > A2,
> > > > > A2 with B1 and B1 with B2.
> > > > >
> > > > > We were not waiting for the major compact after the splits. Thus
it
> > > could
> > > > > be the reason.
> > > > >
> > > > > We will give it a try and come back to you.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Ivan
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 2015-07-15 16:54 GMT+02:00 Jean-Marc Spaggiari <
> > > jean-marc@spaggiari.org
> > > > >:
> > > > >
> > > > >> Hi Ivan,
> > > > >>
> > > > >>
> > > > >> For #1, let's consider regions A, B, C and D all in server S.
> > > > >>
> > > > >> If you merge A and B togher and C and D togher, still in S, you
> will
> > > > have
> > > > >> A' and C' regions only on S.
> > > > >>
> > > > >> Now, if you try to merge again A' and C' into a new bigger region,
> > > this
> > > > >> will fail silently until A' and C' are major compacted.
> > > > >>
> > > > >> Indeed, before major compaction, A' and C' will both still contain
> > > > >> references to A+B and C+D files/regions. So to be able to merge
A'
> > and
> > > > C'
> > > > >> you need this to be cleaned and so you need to major compact
them.
> > > > >>
> > > > >> HTH,
> > > > >>
> > > > >> JM
> > > > >>
> > > > >> 2015-07-15 10:43 GMT-04:00 Ted Yu <yuzhihong@gmail.com>:
> > > > >>
> > > > >> > bq. that some times merge just does not work.
> > > > >> >
> > > > >> > Can you identify under what scenario the merge doesn't work
> > (through
> > > > >> closer
> > > > >> > inspection of the region server log - assuming you have
DEBUG
> > > logging
> > > > >> > turned on) ?
> > > > >> >
> > > > >> > bq. Are there minimum requirements for two regions to be
merged?
> > > > >> >
> > > > >> > If the two adjacent regions to be merged are on different
> servers,
> > > > HBase
> > > > >> > would move one of them to the other region's server.
> > > > >> >
> > > > >> > bq. If those requirements are not met, does the merge process
> exit
> > > > >> > silently?
> > > > >> >
> > > > >> > Nothing I am aware of.
> > > > >> >
> > > > >> > Cheers
> > > > >> >
> > > > >> > On Wed, Jul 15, 2015 at 3:08 AM, Ivan Brondino <
> > ibrondino@fi.upm.es
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi,
> > > > >> > >
> > > > >> > > We are using Hbase 0.98.6 and Hadoop 2.5.0 - cdh 5.3.5.
> > > > >> > >
> > > > >> > > We have a couple of questions regarding splits and
merge:
> > > > >> > >
> > > > >> > > 1) Since both split and merge are async invocations
on the
> > client
> > > > >> side,
> > > > >> > we
> > > > >> > > have a naîve
> > > > >> > > workaround to make it synchronous that is to look at
the RS
> Info
> > > > >> Server
> > > > >> > via
> > > > >> > > an HttpClient checking
> > > > >> > > for pending tasks to be finished. Once all tasks are
finished
> in
> > > all
> > > > >> RS
> > > > >> > we
> > > > >> > > issue the next split/merge for the same table. Is there
a
> better
> > > way
> > > > >> to
> > > > >> > do
> > > > >> > > it?
> > > > >> > >
> > > > >> > > 2) Regarding merges, we've noticed that some times
merge just
> > does
> > > > not
> > > > >> > > work. We see on the master log that merge request is
issued
> and
> > it
> > > > >> > forwards
> > > > >> > > it to the corresponding RS but then on the RS nothing
happens.
> > We
> > > > >> always
> > > > >> > > merge two regions on the same region server.
> > > > >> > > Are there minimum requirements for two regions to be
merged?
> If
> > > > those
> > > > >> > > requirements are not met, does the merge process exit
> silently?
> > > > >> > >
> > > > >> > > Thanks in advance,
> > > > >> > >
> > > > >> > > Iván
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message