crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-147) Pipeline.writeTextFile should work with Avro data
Date Tue, 22 Jan 2013 20:14:13 GMT


Josh Wills updated CRUNCH-147:

    Attachment: CRUNCH-147b.patch

Here's my take, Dave. It fixes the integration tests and simply forces writeTextFile to convert
its input PCollection into strings. The tests are updated for cases where the input PCollection
is a PTable and thus need to expect Pair<K, V>.toString() formatting of the data. What
do you think?
> Pipeline.writeTextFile should work with Avro data
> -------------------------------------------------
>                 Key: CRUNCH-147
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.4.0
>            Reporter: Dave Beech
>            Assignee: Josh Wills
>            Priority: Minor
>         Attachments: CRUNCH-147b.patch, CRUNCH-147.patch
> I'd find it useful to be able to dump Avro collections to a text file (as json) during
development to have a quick look inside without having to mess around with avro-tools. The
current Pipeline.writeTextFile method expects Writables and gives the following error when
you attempt to pass Avro to it:
> Exception in thread "main" java.lang.ClassCastException: class my.avro.Record
> 	at java.lang.Class.asSubclass(
> 	at org.apache.crunch.types.writable.Writables.records(
> 	at org.apache.crunch.types.writable.WritableTypeFamily.records(
> 	at org.apache.crunch.types.PTypeUtils.convert(
> 	at

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message