From dev-return-4262-apmail-giraph-dev-archive=giraph.apache.org@giraph.apache.org Fri Sep 21 18:04:38 2012 Return-Path: X-Original-To: apmail-giraph-dev-archive@www.apache.org Delivered-To: apmail-giraph-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0F261D340 for ; Fri, 21 Sep 2012 18:04:38 +0000 (UTC) Received: (qmail 21003 invoked by uid 500); 21 Sep 2012 18:04:37 -0000 Delivered-To: apmail-giraph-dev-archive@giraph.apache.org Received: (qmail 20932 invoked by uid 500); 21 Sep 2012 18:04:37 -0000 Mailing-List: contact dev-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@giraph.apache.org Delivered-To: mailing list dev@giraph.apache.org Delivered-To: moderator for dev@giraph.apache.org Received: (qmail 690 invoked by uid 99); 21 Sep 2012 17:57:05 -0000 Content-Type: multipart/alternative; boundary="===============8113310130981244114==" MIME-Version: 1.0 Subject: Re: Review Request: GIRAPH-329: Improve network performance From: "Eugene Koontz" To: "Eugene Koontz" , "Avery Ching" , "giraph" Date: Fri, 21 Sep 2012 17:57:04 -0000 Message-ID: <20120921175704.31227.41111@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org Auto-Submitted: auto-generated Sender: "Eugene Koontz" X-ReviewGroup: giraph X-ReviewRequest-URL: https://reviews.apache.org/r/7206/ X-Sender: "Eugene Koontz" References: <20120921081236.4009.85211@reviews.apache.org> In-Reply-To: <20120921081236.4009.85211@reviews.apache.org> Reply-To: "Eugene Koontz" --===============8113310130981244114== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7206/#review11788 ----------------------------------------------------------- Avery, this seems like a similar situation as GIRAPH-262/GIRAPH-289, where = we can break out a separate JIRA for subclassing Configuration. GIRAPH-329 = would depend on that one. http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/girap= h/GiraphConfiguration.java Wouldn't it be good to split this out into another JIRA ("Specialize or= g.apache.hadoop.conf.Configuration to GiraphConfiguration")? http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/girap= h/benchmark/PageRankBenchmark.java I like this improvement! Much cleaner and easier to read, and less inti= midating for beginners. - Eugene Koontz On Sept. 21, 2012, 8:12 a.m., Avery Ching wrote: > = > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7206/ > ----------------------------------------------------------- > = > (Updated Sept. 21, 2012, 8:12 a.m.) > = > = > Review request for giraph. > = > = > Description > ------- > = > Major changes: > - Configuration have been deprecated in favor of GiraphConfiguration for = easy get/set methods. Then I have ImmutableClassesGiraphConfiguration that= will generate the classes as final variables to be thread-safe. This is a= big improvement over Configuration#getClass(), which is very slow and make= s deserialization very slow. This represents the bulk of the changes as we= have to change code everywhere. All configuration variables have moved fr= om GiraphJob to GiraphConfiguration. Note that I didn't fix the vertex inp= ut / output APIs to use ImmutableClassesGiraphConfiguration and just genera= te it there for now. That can be done in a later change. > - There is a new ImmutableClassesGiraphConfigurable that replaces Configu= rable and our ReflectionUtils understands this to make sure to newInstance = an object and set its ImmutableClassesGiraphConfiguration if it is Immutabl= eClassesGiraphConfigurable. > - Upgraded/downgraded log messages to trace/debug to make debug more usab= le. I'm able to run most jobs in debug with about a 10% loss in performanc= e, not too bad. Debug is now quite useful to figuring out performance issu= es, general problems. > - Added an optimization to SimpleMessageStore#addPartitionMessages to avo= id getting the partition map for every vertex. This was one of the bottlen= ecks. > - Added client and server ExecutionHandlers to Netty to avoid doing the b= usiness logic with I/O threads (This used to clog up the pipe) > - Added WrappedAdaptiveReceiveBufferSizePredictorFactory to debug/predict= the size of the incoming messages and provide better performance tuning on= the server instead of receiving a bunch of small messages. > = > Minor changes: > - Added BspUtilsTest to try various ways of testing class creation, perfo= rmance and correctness. > - Upgrade junit from 4.0 -> 4.8 to allow getting the test name (See BspUt= ilsTest.java). > = > = > This addresses bug GIRAPH-329. > https://issues.apache.org/jira/browse/GIRAPH-329 > = > = > Diffs > ----- > = > http://svn.apache.org/repos/asf/giraph/trunk/pom.xml 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/GiraphConfiguration.java PRE-CREATION = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/GiraphRunner.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/ImmutableClassesGiraphConfigurable.java PRE-CREATION = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/ImmutableClassesGiraphConfiguration.java PRE-CREATION = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/benchmark/EdgeListVertexPageRankBenchmark.java PRE-CREATION = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/benchmark/PageRankBenchmark.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/benchmark/RandomMessageBenchmark.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/benchmark/ShortestPathsBenchmark.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/bsp/BspInputFormat.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/BasicRPCCommunications.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/RPCCommunications.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/SendMessageCache.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/ServerData.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/messages/DiskBackedMessageStore.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/messages/SequentialFileMessageStore.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/messages/SimpleMessageStore.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/ByteCounter.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/NettyClient.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/NettyMasterClient.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/NettyMasterClientServer.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/NettyMasterServer.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/NettyServer.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/NettyWorkerClient.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/NettyWorkerClientServer.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/NettyWorkerServer.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/WrappedAdaptiveReceiveBufferSizePredictorFactory.java PRE-= CREATION = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/handler/AddressRequestIdGenerator.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/handler/RequestDecoder.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/handler/RequestEncoder.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/handler/RequestInfo.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/handler/RequestServerHandler.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/handler/ResponseClientHandler.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/netty/handler/WorkerRequestReservedMap.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/requests/SendPartitionMessagesRequest.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/requests/SendPartitionMutationsRequest.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/requests/SendVertexRequest.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/comm/requests/WritableRequest.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/examples/GeneratedVertexReader.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/examples/SimpleCheckpointVertex.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/examples/SimplePageRankVertex.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/examples/SimpleSuperstepVertex.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/examples/SimpleVertexWithWorkerContext.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/BspService.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/BspServiceMaster.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/BspServiceWorker.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/BspUtils.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/EdgeListVertex.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/GiraphJob.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/GraphMapper.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/HashMapVertex.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/MasterCompute.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/MasterThread.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/MutableVertex.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/SimpleMutableVertex.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/SimpleVertex.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/Vertex.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/VertexMutations.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/VertexResolver.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/BasicPartitionOwner.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/DiskBackedPartitionStore.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/GraphPartitionerFactory.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/HashMasterPartitioner.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/HashPartitionerFactory.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/HashRangePartitionerFactory.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/Partition.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/RangePartitionOwner.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/RangeSplitHint.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/graph/partition/SimplePartitionStore.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/integration/SuperstepHashPartitionerFactory.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/io/JsonBase64VertexInputFormat.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/io/PseudoRandomVertexInputFormat.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/io/TextVertexInputFormat.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/utils/FakeTime.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/utils/InternalVertexRunner.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/utils/ReflectionUtils.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/utils/SystemTime.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/utils/Time.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/utils/WritableUtils.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/main/java/org/apache/g= iraph/zk/ZooKeeperManager.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/BspCase.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/TestAggregatorsHandling.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/TestAutoCheckpoint.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/TestBspBasic.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/TestGraphPartitioner.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/TestJsonBase64Format.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/TestManualCheckpoint.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/TestNotEnoughMapTasks.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/TestVertexTypes.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/aggregators/TestDoubleAggregators.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/aggregators/TestFloatAggregators.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/comm/ConnectionTest.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/comm/RPCCommunicationsTest.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/comm/RequestFailureTest.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/comm/RequestTest.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/comm/TestMessageStores.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/examples/SimpleShortestPathsVertexTest.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/examples/TryMultiRpcBindingPortsTest.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/graph/TestEdgeListVertex.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/graph/partition/TestPartitionStores.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/io/TestLongDoubleDoubleAdjacencyListVertexInputFormat.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/io/TestTextDoubleDoubleAdjacencyListVertexInputFormat.java 1388358 = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/org/apache/g= iraph/utils/BspUtilsTest.java PRE-CREATION = > http://svn.apache.org/repos/asf/giraph/trunk/src/test/java/zk/TestZooKe= eperManager.java 1388358 = > = > Diff: https://reviews.apache.org/r/7206/diff/ > = > = > Testing > ------- > = > Passed all unittests, ran PageRankBenchmark many times on a real cluster.= See results in https://issues.apache.org/jira/secure/attachment/12546012/= GIRAPH-329.patch . > = > = > Thanks, > = > Avery Ching > = > --===============8113310130981244114==--