giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maja Kabiljo" <majakabi...@fb.com>
Subject Re: Review Request: GIRAPH-498: We should check input splits status from zookeeeper once per worker, not once per split thread
Date Tue, 05 Feb 2013 18:34:35 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9301/
-----------------------------------------------------------

(Updated Feb. 5, 2013, 6:34 p.m.)


Review request for giraph.


Changes
-------

useInputSplitLocality


Description
-------

When using a lot of workers and a lot of input split threads, checking that all input splits
are finished after the reading is done takes a long time, since we check every input split
once per thread.


This addresses bug GIRAPH-498.
    https://issues.apache.org/jira/browse/GIRAPH-498


Diffs (updated)
-----

  giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 796047d 
  giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java a48c5ea 
  giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java 7d40dfb

  giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallableFactory.java 1adcd73

  giraph-core/src/main/java/org/apache/giraph/worker/InputSplitPathOrganizer.java bfaefd2

  giraph-core/src/main/java/org/apache/giraph/worker/InputSplitsCallable.java d09ca2b 
  giraph-core/src/main/java/org/apache/giraph/worker/InputSplitsHandler.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java a4f98e1

  giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallableFactory.java
0d617dc 
  giraph-core/src/test/java/org/apache/giraph/TestBspBasic.java 987f51c 

Diff: https://reviews.apache.org/r/9301/diff/


Testing
-------

mvn clean verify

Real application, using 200 workers and 20 input threads:
- trunk - about 560s for input split threads to finish, 720s for input superstep
- with this patch - about 310s for input split threads to finish, 500s for input superstep


Thanks,

Maja Kabiljo


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message