giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Avery Ching (JIRA)" <>
Subject [jira] [Commented] (GIRAPH-329) Improve network performance
Date Fri, 21 Sep 2012 08:15:07 GMT


Avery Ching commented on GIRAPH-329:

Actually, I've decided that we might not need to split it.  95% of the changes are related
to changing configuration to be Giraph specific (inspired by GIRAPH-214).  The remaining changes
are quite simple and easy to understand.

Major changes:
- Configuration have been deprecated in favor of GiraphConfiguration for easy get/set methods.
 Then I have ImmutableClassesGiraphConfiguration that will generate the classes as final variables
to be thread-safe.  This is a big improvement over Configuration#getClass(), which is very
slow and makes deserialization very slow.  This represents the bulk of the changes as we have
to change code everywhere.  All configuration variables have moved from GiraphJob to GiraphConfiguration.
 Note that I didn't fix the vertex input / output APIs to use ImmutableClassesGiraphConfiguration
and just generate it there for now.  That can be done in a later change.
- There is a new ImmutableClassesGiraphConfigurable that replaces Configurable and our ReflectionUtils
understands this to make sure to newInstance an object and set its ImmutableClassesGiraphConfiguration
if it is ImmutableClassesGiraphConfigurable.
- Upgraded/downgraded log messages to trace/debug to make debug more usable.  I'm able to
run most jobs in debug with about a 10% loss in performance, not too bad.  Debug is now quite
useful to figuring out performance issues, general problems.
- Added an optimization to SimpleMessageStore#addPartitionMessages to avoid getting the partition
map for every vertex.  This was one of the bottlenecks.
- Added client and server ExecutionHandlers to Netty to avoid doing the business logic with
I/O threads (This used to clog up the pipe)
- Added WrappedAdaptiveReceiveBufferSizePredictorFactory to debug/predict the size of the
incoming messages and provide better performance tuning on the server instead of receiving
a bunch of small messages.

Minor changes:
- Added BspUtilsTest to try various ways of testing class creation, performance and correctness.
- Upgrade junit from 4.0 -> 4.8 to allow getting the test name (See
> Improve network performance
> ---------------------------
>                 Key: GIRAPH-329
>                 URL:
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Avery Ching
>            Assignee: Avery Ching
>         Attachments: GIRAPH-329.patch, nettyImprovement.png, NettyImprovements.xlsx
> I have been working on some network improvements that appear to give 1.5x to 2x.  Unfortunately,
this is a very large change and very hard to split out.  Will give an update soon.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message