crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chao Shi (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-308) Upgrade to Hadoop 2.2.0 and HBase 0.96
Date Mon, 09 Dec 2013 02:42:07 GMT


Chao Shi commented on CRUNCH-308:

bq. are there some row keys that have so many values that sorting them in the reducer uses
too much memory?
I don't know why they have to sort them in reducer. I guess this is because of the difficulty
to wire up complex pipeline with plain MR (e.g. with Crunch, we can easily sort KVs of each
family independently). Your second part looks good. Thanks!

> Upgrade to Hadoop 2.2.0 and HBase 0.96
> --------------------------------------
>                 Key: CRUNCH-308
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>            Reporter: Josh Wills
>         Attachments: CRUNCH-308.patch, CRUNCH-HBASE96.patch
> As discussed on dev@crunch, we should update Crunch to run against the new mainline releases
of Hadoop (2.2.0) and HBase (0.96).
> There isn't a good way to maintain a shim between HBase 0.94 and HBase 0.96 due to a
number of API changes, so this change means that support for HBase 0.94 will remain in the
0.8.x sequence of Crunch releases, and 0.96 will be the supported version from 0.9.0 onwards.

This message was sent by Atlassian JIRA

View raw message