samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edi Bice <edi_b...@yahoo.com>
Subject Re: Review Request 43732: Implemented AvroDataFileHdfsWriter
Date Wed, 02 Mar 2016 16:35:41 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/43732/
-----------------------------------------------------------

(Updated March 2, 2016, 4:35 p.m.)


Review request for samza.


Changes
-------

Rebased with latest from master branch as requested


Repository: samza


Description
-------

https://issues.apache.org/jira/browse/SAMZA-876 

Implemented AvroDataFileHdfsWriter fashioned loosely after BinarySequenceFileHDFSWriter.


Diffs (updated)
-----

  docs/learn/documentation/versioned/hdfs/producer.md cfd22c6 
  docs/learn/documentation/versioned/jobs/configuration-table.html 6705530 
  gradle/dependency-versions.gradle 52e25aa 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsConfig.scala 7993119 
  samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/AvroDataFileHdfsWriter.scala
PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-batch-job-avro.properties PRE-CREATION 
  samza-hdfs/src/test/resources/samza-hdfs-test-job-avro.properties PRE-CREATION 
  samza-hdfs/src/test/scala/org/apache/samza/system/hdfs/TestHdfsSystemProducerTestSuite.scala
c4b04a1 

Diff: https://reviews.apache.org/r/43732/diff/


Testing
-------

Two JUnit tests similar to the Text/BinarySequenceFileHdfsWriter ones. In addition I've been
using AvroDataFileHdfsWriter at the end of my pipeline. I feed the generated avro files to
Apache Samoa. Have processed millions of records successfully.


Thanks,

Edi Bice


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message