hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doğacan Güney <doga...@gmail.com>
Subject Re: MapReduce in hbase 0.20
Date Tue, 07 Jul 2009 20:26:42 GMT
On Tue, Jul 7, 2009 at 19:52, stack <stack@duboce.net> wrote:

> 2009/7/7 Doğacan Güney <dogacan@gmail.com>
>
> > ...
> > > What would you suggest Doğacan?  Maybe we should add Marker interfaces
> to
> > > Put and Delete and then change TableReducer to take the Marker?
> > >
> >
> > Sure, that's a good idea.
> >
> > I haven't studied hadoop 0.20's API much yet so I am not sure if this can
> > be
> > done but can hbase have its own ReduceContext class? If this is possible,
> > then maybe we can just expose the HTable instance through the context and
> > allow user to do whatever he wants to do on the table (and throw an
> > exception if context.write is called) . I think this would be much more
> > simpler to understand than the write/collect() calls (e.g
> TableOutputFormat
> > ignores the collect-ed keys). Does this make sense?
>
>
> This is an interesting idea.  Sketch more how it work.  Currently the
> HTable
> is made in the TableOutputWriter.  Would we then change the Reducer input
> so
> it took any Writable rather than IBW and Put?
>

Yes, so TableReducer would look something like this:

class TableReducer<K extends WritableComparable<?>, V extends Writable>
extends Reducer<K, V, WritableComparable, Writable> { /* ..... */ };

And in general, you would write your reduce method like this:

void reduce(K, V, HbaseContext context) {
    Put put = new Put(row);
    // ...........
    // add stuff to put
    // ...........
    Delete delete = new Delete(row);
    // ......
    // add stuff to delete
    // .......

    context.putToTable(put);
    context.deleteFromTable(delete);
    // and context.write may raise an exception
}

Anyway, I guess we are close to 0.20 release. So if this is unnecessarily
complex, the solution you described works just fine for me.


>
> St.Ack
>



-- 
Doğacan Güney

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message