giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Armando" <a.mirag...@student.vu.nl>
Subject Review Request 13756: GIRAPH-732: EdgeOutputFormat API
Date Fri, 23 Aug 2013 01:34:47 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13756/
-----------------------------------------------------------

Review request for giraph, Armando, Claudio Martella, and Nitay Joffe.


Bugs: GIRAPH-732
    https://issues.apache.org/jira/browse/GIRAPH-732


Repository: giraph-git


Description
-------

This patch is a possible solution to provide an EdgeOutputFormat API in Giraph.
The idea is to add the same logic as the one for the vertices for edges. For this reason I
add saveEdges after saveVertices.
This requires the user to have to different paths for the resulting files. For this reason
I have implemented TextEdgeOutputFormat
and modified TextVertexOutputFormat so that a subdirectory can be specified.
The code I provide should be compliant with previous versions of hadoop as well as YARN (I
was yet not able to test it on YARN).
It is also retro-compatible since when not specified, TextVertexOutputFormat behaves as before.
I am also providing an actual usable implementation with the associated tests (SrcIdDstIdEdgeValueOutputFormat).
Also very interesting is the fact that this implementation is totally transparent to the classes
implementing TextVertexOutputFormat.

passed "maven verify"


Diffs
-----

  giraph-core/src/main/java/org/apache/giraph/GiraphRunner.java 1bd79b5 
  giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java 71fe885 
  giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 23bcd32 
  giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java c276c2a 
  giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
49a2ebc 
  giraph-core/src/main/java/org/apache/giraph/io/EdgeOutputFormat.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/io/EdgeWriter.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/io/formats/GiraphTextOutputFormat.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/io/formats/SrcIdDstIdEdgeValueTextOutputFormat.java
PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/io/formats/TextEdgeOutputFormat.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/io/formats/TextVertexOutputFormat.java c91d543

  giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedEdgeOutputFormat.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/utils/ConfigurationUtils.java 745764b 
  giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java da1e7fb 
  giraph-core/src/test/java/org/apache/giraph/io/TestSrcIdDstIdEdgeValueTextOutputFormat.java
PRE-CREATION 

Diff: https://reviews.apache.org/r/13756/diff/


Testing
-------


Thanks,

Armando


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message