From dev-return-3571-apmail-crunch-dev-archive=crunch.apache.org@crunch.apache.org Sun Dec 1 21:47:35 2013 Return-Path: X-Original-To: apmail-crunch-dev-archive@www.apache.org Delivered-To: apmail-crunch-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3587B10CC0 for ; Sun, 1 Dec 2013 21:47:35 +0000 (UTC) Received: (qmail 30386 invoked by uid 500); 1 Dec 2013 21:47:35 -0000 Delivered-To: apmail-crunch-dev-archive@crunch.apache.org Received: (qmail 30317 invoked by uid 500); 1 Dec 2013 21:47:35 -0000 Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@crunch.apache.org Delivered-To: mailing list dev@crunch.apache.org Received: (qmail 30308 invoked by uid 500); 1 Dec 2013 21:47:35 -0000 Delivered-To: apmail-incubator-crunch-dev@incubator.apache.org Received: (qmail 30302 invoked by uid 99); 1 Dec 2013 21:47:35 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Dec 2013 21:47:35 +0000 Date: Sun, 1 Dec 2013 21:47:35 +0000 (UTC) From: "Josh Wills (JIRA)" To: crunch-dev@incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CRUNCH-308) Upgrade to Hadoop 2.2.0 and HBase 0.96 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CRUNCH-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Wills updated CRUNCH-308: ------------------------------ Attachment: CRUNCH-HBASE96.patch Patch to get this working-- big changes are in dependencies (protobuf 2.5.0 to work with Hadoop 2.2.0) and all of the HBase changes. Most of the work was in creating new Writable-based PTypes for HBase (since Put/Delete/KeyValue are no longer Writables), and some changes to the HFileOutputFormatForCrunch to ensure that partition-based sorting still works correctly (switched the key type to be ImmutableBytesWritable instead of KeyValue for the sorting.) > Upgrade to Hadoop 2.2.0 and HBase 0.96 > -------------------------------------- > > Key: CRUNCH-308 > URL: https://issues.apache.org/jira/browse/CRUNCH-308 > Project: Crunch > Issue Type: Bug > Reporter: Josh Wills > Attachments: CRUNCH-HBASE96.patch > > > As discussed on dev@crunch, we should update Crunch to run against the new mainline releases of Hadoop (2.2.0) and HBase (0.96). > There isn't a good way to maintain a shim between HBase 0.94 and HBase 0.96 due to a number of API changes, so this change means that support for HBase 0.94 will remain in the 0.8.x sequence of Crunch releases, and 0.96 will be the supported version from 0.9.0 onwards. -- This message was sent by Atlassian JIRA (v6.1#6144)