crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-308) Upgrade to Hadoop 2.2.0 and HBase 0.96
Date Sun, 01 Dec 2013 21:47:35 GMT


Josh Wills updated CRUNCH-308:

    Attachment: CRUNCH-HBASE96.patch

Patch to get this working-- big changes are in dependencies (protobuf 2.5.0 to work with Hadoop
2.2.0) and all of the HBase changes. Most of the work was in creating new Writable-based PTypes
for HBase (since Put/Delete/KeyValue are no longer Writables), and some changes to the HFileOutputFormatForCrunch
to ensure that partition-based sorting still works correctly (switched the key type to be
ImmutableBytesWritable instead of KeyValue for the sorting.)

> Upgrade to Hadoop 2.2.0 and HBase 0.96
> --------------------------------------
>                 Key: CRUNCH-308
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>            Reporter: Josh Wills
>         Attachments: CRUNCH-HBASE96.patch
> As discussed on dev@crunch, we should update Crunch to run against the new mainline releases
of Hadoop (2.2.0) and HBase (0.96).
> There isn't a good way to maintain a shim between HBase 0.94 and HBase 0.96 due to a
number of API changes, so this change means that support for HBase 0.94 will remain in the
0.8.x sequence of Crunch releases, and 0.96 will be the supported version from 0.9.0 onwards.

This message was sent by Atlassian JIRA

View raw message