giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavan Kumar Athivarapu" <pava...@outlook.com>
Subject Re: Review Request 22234: support for partitioned input in giraph
Date Thu, 05 Jun 2014 06:47:18 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22234/
-----------------------------------------------------------

(Updated June 5, 2014, 6:47 a.m.)


Review request for giraph, Avery Ching, Sergey Edunov, Igor Kabiljo, and Maja Kabiljo.


Changes
-------

address comments.


Repository: giraph-git


Description
-------

There are many changes here:

A new input format - MappingInputFormat - and related dependencies has been defined
New hive input format classes to read Mapping table + some examples which can be used for
running hellopagerank with modifications in test plan have been defined
Changes to main giraph classes to read the mapping & use it for getting partition info
- in worker, master & partition sections
New mapping section defined to define & declare MappingStore format & some sample
implementations
The code can take 2 paths based on what the user wants

Embed info into vertexId by implementing proper contracts (TranslateEdge.interface + 2 methods
in MappingStore)
then use an EmbeddedGraphPartitioner to directly read worker info off of vertex ids [translate
once & user freely for the rest of app]

Always read from the MappingStore (this has more overhead probably because of cache misses
in processor) [never translate but pay cost of map lookup each time partition / worker info
is needed]


Diffs (updated)
-----

  giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java ec0ddbb 
  giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java bda967d 
  giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java 3337621 
  giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java 6b36418 
  giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
95e029d 
  giraph-core/src/main/java/org/apache/giraph/io/MappingInputFormat.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/io/MappingReader.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedMappingInputFormat.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedMappingReader.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/io/iterables/MappingReaderWrapper.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/mapping/AbstractLongByteOps.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/mapping/DefaultEmbeddedLongByteOps.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/mapping/DefaultLongByteOps.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/mapping/LongByteMappingStore.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/mapping/MappingEntry.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/mapping/MappingStore.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/mapping/MappingStoreOps.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/mapping/package-info.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/mapping/translate/LongByteTranslateEdge.java
PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/mapping/translate/TranslateEdge.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/mapping/translate/package-info.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 90dc9f3 
  giraph-core/src/main/java/org/apache/giraph/master/MasterThread.java 15dbe07 
  giraph-core/src/main/java/org/apache/giraph/partition/GraphPartitionerFactory.java 4200d79

  giraph-core/src/main/java/org/apache/giraph/partition/HashPartitionerFactory.java 7cc5651

  giraph-core/src/main/java/org/apache/giraph/partition/HashRangePartitionerFactory.java 1eeece7

  giraph-core/src/main/java/org/apache/giraph/partition/LongMappingStorePartitionerFactory.java
PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/partition/SimpleIntRangePartitionerFactory.java
8ab692f 
  giraph-core/src/main/java/org/apache/giraph/partition/SimpleLongRangePartitionerFactory.java
2989598 
  giraph-core/src/main/java/org/apache/giraph/partition/SimplePartitionerFactory.java 15b0756

  giraph-core/src/main/java/org/apache/giraph/partition/SimpleWorkerPartitioner.java 600d7a3

  giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java aff7084 
  giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java 828eac4

  giraph-core/src/main/java/org/apache/giraph/worker/FullInputSplitCallable.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/worker/LocalData.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/worker/MappingInputSplitsCallable.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/worker/MappingInputSplitsCallableFactory.java
PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java e3e04d6

  giraph-core/src/test/java/org/apache/giraph/partition/SimpleRangePartitionFactoryTest.java
4e19cd2 
  giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java 603910b 
  giraph-hive/src/main/java/org/apache/giraph/hive/common/GiraphHiveConstants.java c7ad63b

  giraph-hive/src/main/java/org/apache/giraph/hive/common/HiveUtils.java 2388673 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/AbstractHiveToMapping.java
PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveMappingInputFormat.java
PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveMappingReader.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveToMapping.java PRE-CREATION

  giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/SimpleHiveToMapping.java
PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/LongByteHiveToMapping.java
PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/LongInt2ByteHiveToMapping.java
PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/package-info.java
PRE-CREATION 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/package-info.java PRE-CREATION


Diff: https://reviews.apache.org/r/22234/diff/


Testing
-------

ran pagerank jobs multiple times
mvn clean verify


Thanks,

Pavan Kumar Athivarapu


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message