hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Neumann <neun...@gmail.com>
Subject Re: hadoop without append in the absence of puts
Date Thu, 23 Jun 2011 02:22:38 GMT
Andy,
we will use LoadIncrementalHFiles, are you saying that this will never cause
a split?

Joey,
our policies will not allow us to deploy a Hadoop version with append, until
there is an Apache release that includes both security and append. So
t.isis not about cost of appending to the WAL - we simply don't have
that
option.

Thanks -Andreas.

On Wed, Jun 22, 2011 at 7:14 PM, Joey Echeverria <joey@cloudera.com> wrote:

> > I agree that even though these are rare, they are not rare enough to take
> a
> > risk. But they could be rare enough to justify a less efficient
> > implementation of the WAL. Would it be reasonable to use an
> implementation
> > of HLog that - at the price of performance - persists the WAL to HDFS
> > without relying on append?
>
> Since edits to .META. should be rare if you're using bulkImport, why
> not just pay the cost of appending to the WAL? My guess is performance
> would not suffer greatly. Would be nice to benchmark my claim :)
>
> -Joey
>

On Wed, Jun 22, 2011 at 7:09 PM, Andrew Purtell <apurtell@apache.org> wrote:

> > From: Andreas Neumann <neunand@gmail.com>
>
> > I guess that includes region splits but also
> > reassignment of a region after its region server died.
>
> If you are not writing, then you won't see splits.
>
> Reassignment does involve writes to META but missing these is less serious
> than missing a change in table structure. Stack could say more here but I
> believe the Master will discover inconsistency and repair it.
>
> > But they could be rare enough to justify a less efficient
> > implementation of the WAL. Would it be reasonable to use an
> > implementation of HLog that - at the price of performance -
> > persists the WAL to HDFS without relying on append?
>
> How would that work if not create/write/close on every WAL commit? Correct
> me if I'm wrong, but there is no way to guarantee persistence in an HDFS
> without append except to .close() the file.
>
> HBase can do the (near) equivalent of this without requiring any code
> changes. Set hbase.regionserver.logroll.period to an insanely low value.
>
>   - Andy
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message