crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias Friedrich" <m...@mafr.de>
Subject Re: Review Request: Update the io.* package to support all InputFormats that ship with hadoop-client.
Date Mon, 03 Dec 2012 17:16:55 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8308/#review13975
-----------------------------------------------------------

Ship it!


Looks good! Some nice to have suggestions below, but +1 for commit in any case.


crunch/src/main/java/org/apache/crunch/io/text/LineParser.java
<https://reviews.apache.org/r/8308/#comment29895>

    What about something like Splitter.on(sep).limit(1).split(input)?



crunch/src/main/java/org/apache/crunch/io/text/NLineFileSource.java
<https://reviews.apache.org/r/8308/#comment29896>

    I think the linesPerTask parameter is interesting enough to warrant some javadoc.


- Matthias Friedrich


On Dec. 2, 2012, 10:14 p.m., Josh Wills wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8308/
> -----------------------------------------------------------
> 
> (Updated Dec. 2, 2012, 10:14 p.m.)
> 
> 
> Review request for crunch.
> 
> 
> Description
> -------
> 
> We should add support for the NLineInputFormat and the KeyValueTextInputFormat so that
all of the InputFormats in hadoop's mapredice.lib.input package are supported in Crunch.
> 
> 
> This addresses bug CRUNCH-119.
>     https://issues.apache.org/jira/browse/CRUNCH-119
> 
> 
> Diffs
> -----
> 
>   crunch/src/it/java/org/apache/crunch/io/CompositePathIterableIT.java 796b821 
>   crunch/src/it/java/org/apache/crunch/io/NLineInputIT.java PRE-CREATION 
>   crunch/src/it/java/org/apache/crunch/io/TextFileTableIT.java PRE-CREATION 
>   crunch/src/main/java/org/apache/crunch/io/ReadableSource.java 73a13a3 
>   crunch/src/main/java/org/apache/crunch/io/avro/AvroFileReaderFactory.java 6f21dd2 
>   crunch/src/main/java/org/apache/crunch/io/avro/AvroFileSource.java 2226556 
>   crunch/src/main/java/org/apache/crunch/io/impl/AutoClosingIterator.java d58f290 
>   crunch/src/main/java/org/apache/crunch/io/impl/FileTableSourceImpl.java f6e8f1d 
>   crunch/src/main/java/org/apache/crunch/io/seq/SeqFileReaderFactory.java ad1b81b 
>   crunch/src/main/java/org/apache/crunch/io/seq/SeqFileSource.java e8f3dcf 
>   crunch/src/main/java/org/apache/crunch/io/seq/SeqFileTableReaderFactory.java 20c749a

>   crunch/src/main/java/org/apache/crunch/io/seq/SeqFileTableSource.java 56ed985 
>   crunch/src/main/java/org/apache/crunch/io/text/LineParser.java PRE-CREATION 
>   crunch/src/main/java/org/apache/crunch/io/text/NLineFileSource.java PRE-CREATION 
>   crunch/src/main/java/org/apache/crunch/io/text/TextFileReaderFactory.java a0c48e0 
>   crunch/src/main/java/org/apache/crunch/io/text/TextFileSource.java ee51c04 
>   crunch/src/main/java/org/apache/crunch/io/text/TextFileTableSource.java PRE-CREATION

>   crunch/src/main/java/org/apache/crunch/io/text/TextFileTableSourceTarget.java PRE-CREATION

>   crunch/src/main/java/org/apache/crunch/io/text/TextFileTarget.java c7e06d3 
>   crunch/src/test/java/org/apache/crunch/io/avro/AvroFileReaderFactoryTest.java 66863ba

> 
> Diff: https://reviews.apache.org/r/8308/diff/
> 
> 
> Testing
> -------
> 
> Unit and integration on the new sources.
> 
> 
> Thanks,
> 
> Josh Wills
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message