crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-308) Upgrade to Hadoop 2.2.0 and HBase 0.96
Date Fri, 06 Dec 2013 14:37:36 GMT


Josh Wills commented on CRUNCH-308:

Ah, I see what you're saying. I couldn't figure out a way to make sorting KeyValues as BytesWritables
work in HBase 0.96, although I can take another crack at it if you think it's important enough.
I'd like to understand why the in-memory sorting in KeyValueSortReducer isn't good enough
for our needs; are there some row keys that have so many values that sorting them in the reducer
uses too much memory?

> Upgrade to Hadoop 2.2.0 and HBase 0.96
> --------------------------------------
>                 Key: CRUNCH-308
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>            Reporter: Josh Wills
>         Attachments: CRUNCH-HBASE96.patch
> As discussed on dev@crunch, we should update Crunch to run against the new mainline releases
of Hadoop (2.2.0) and HBase (0.96).
> There isn't a good way to maintain a shim between HBase 0.94 and HBase 0.96 due to a
number of API changes, so this change means that support for HBase 0.94 will remain in the
0.8.x sequence of Crunch releases, and 0.96 will be the supported version from 0.9.0 onwards.

This message was sent by Atlassian JIRA

View raw message