hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hong Tang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6381) WritableUtils::*VLong utilities should be available for byte arrays
Date Fri, 20 Nov 2009 03:30:39 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12780413#action_12780413

Hong Tang commented on HADOOP-6381:

bq. On the other hand, we've lived without a set of static, byte[]-oriented DataInput functions
that would admit identical arguments. If RawComparators are an advanced feature (in need of
better documentation?) or something better handled transparently by Avro, Jute, et al. then
perhaps the user that would appreciate these doesn't exist.

Agreed. I always feel that we lack a cohesive story in terms of raw-comparator (together with
writable). For example, how do I write a raw comparator for a composite key where each field
in the key defines a raw comparator.  protocol buffer would solve it easily because it does
length prefixed encoding and is able to skip fields at any level. I think avro has the potential
of untangling the mess. So I would also like to discourage the introduction of such ad-hoc
apis in io.WritableUtil.

Could we first add these utility methods in the module where it is needed? And possibly migrate
them to the io package when we see more usage cases? This is how BoundedByteArrayOutputStream
is migrated from tfile to io.

Alternatively, if you feel WritableUtil is the right place, we probably should document it
clearly that it should be limited in writing raw comparators and avro may obsolete such apis
in the future.

> WritableUtils::*VLong utilities should be available for byte arrays
> -------------------------------------------------------------------
>                 Key: HADOOP-6381
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6381
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: util
>            Reporter: Chris Douglas
>            Priority: Minor
>         Attachments: C6381-0.patch, C6381-1.patch, C6381-2.patch
> Particularly when working with raw bytes in Writables, it is often useful to have versions
of the vint utility functions for byte arrays.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message