hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Demai Ni <nid...@gmail.com>
Subject Re: no-flush based snapshot policy?
Date Thu, 03 Apr 2014 16:26:35 GMT
Ted,

you are right. We are targeting HBASE-7912 to 1.0 and 0.98, which 1.0 as
the priority for now.  :-)

BTW, we have some code to leverage HBASE-9426 so that we can go distributed
LOG roll at RS level before taking snapshot. I will open a jira to share
that code for discussion purpose.

Demai


On Wed, Apr 2, 2014 at 6:19 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> HBASE-9426 (Make custom distributed barrier procedure pluggable) has been
> back ported to 0.98
> So porting the work from HBASE-7912 to 0.98 would be relatively easy.
>
> I am not aware of this going into 0.94
>
> Cheers
>
>
> On Wed, Apr 2, 2014 at 6:13 PM, Varun Sharma <varun@pinterest.com> wrote:
>
> > Seems like those JIRAs are 1.0 - did not see a 0.94 version # there ?
> >
> >
> > On Wed, Apr 2, 2014 at 1:40 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >
> > > Tianying:
> > > Have you seen the design doc attached to HBASE-7912 'HBase
> Backup/Restore
> > > Based on HBase Snapshot' ?
> > >
> > > Cheers
> > >
> > > >
> > > > > On Tue, Mar 25, 2014 at 2:38 PM, Tianying Chang <tychang@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I need a new snapshot policy. Basically, I cannot disable the
> > table,
> > > > but
> > > > > I
> > > > > > also don't need the snapshot to be that "consistent" where all
RS
> > > > > > coordinated to flush the region before taking the snapshot since
> it
> > > > slow
> > > > > > down production cluster when flush take too long. It is OK for
me
> > if
> > > > the
> > > > > > snapshot missed the data in memstore because I will use WALPlayer
> > to
> > > > fill
> > > > > > the data gap that is not in the snapshot but has been persisted
> (in
> > > > WAL).
> > > > > > So I should have no data loss.
> > > > > >
> > > > > > As a quick hack way to test this in my hbase backup workflow,
I
> > just
> > > > add
> > > > > a
> > > > > > config key, and skip the flushcache() in file
> > > > > > *regionserver/snapshot/FlushSnapshotSubprocedure.java*, something
> > > like
> > > > > > below.  It seems works fine for me, where all data are recovered
> > in a
> > > > new
> > > > > > cluster after running WALPlayer.
> > > > > >
> > > > > > Does anyone see any problem like data corruption, etc?
> > > > > >
> > > > > >
> > > > > > LOG.debug("Flush Snapshotting region " + region.toString() +
"
> > > > > > started...");
> > > > > > if (noFlushNeeded)
> > > > > > {
> > > > > >    LOG.debug("No flush before taking snapshot");
> > > > > > } else
> > > > > > {
> > > > > >     region.flushcache();
> > > > > > }
> > > > > >
> > > > > > If there is no data corruption issue with this policy, I can
add
> an
> > > > > > parameter from hbase shell, so that people can dynamically decide
> > > when
> > > > to
> > > > > > use no-flush snapshot.
> > > > > >
> > > > > > Thanks
> > > > > > Tian-Ying
> > > > > >
> > > > > > On Tue, Mar 25, 2014 at 2:08 PM, Tianying Chang <
> tychang@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I need a new snapshot policy which sits in between the
disabled
> > and
> > > > > > > flushed version. So, basically:
> > > > > > > I cannot disable the table, but I also don't need the snapshot
> to
> > > be
> > > > > that
> > > > > > > "consistent" where all RS coordinated to flush the region
> before
> > > > taking
> > > > > > the
> > > > > > > snapshot.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message