giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Avery Ching (JIRA)" <j...@apache.org>
Subject [jira] [Created] (GIRAPH-12) Investigate communication improvements
Date Mon, 29 Aug 2011 03:11:42 GMT
Investigate communication improvements
--------------------------------------

                 Key: GIRAPH-12
                 URL: https://issues.apache.org/jira/browse/GIRAPH-12
             Project: Giraph
          Issue Type: Improvement
            Reporter: Avery Ching
            Priority: Minor


Currently every worker will start up a thread to communicate with every other workers.  Hadoop
RPC is used for communication.  For instance if there are 400 workers, each worker will create
400 threads.  This ends up using a lot of memory, even with the option  

-Dmapred.child.java.opts="-Xss64k".  

It would be good to investigate using frameworks like Netty or custom roll our own to improve
this situation.  By moving away from Hadoop RPC, we would also make compatibility of different
Hadoop versions easier.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message