giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claudio Martella (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-45) Improve the way to keep outgoing messages
Date Sat, 26 Nov 2011 19:31:40 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157588#comment-13157588
] 

Claudio Martella commented on GIRAPH-45:
----------------------------------------

was thinking, what about storing the out-of-core messages with leveldb?

writing them as they come should be quite efficient as it writes data to an append-only commit
log.
it keeps data sorted, so reading with an iterator should be very fast.

each worker could have for each superstep a leveldb database and each message could have a
key composed like this: <destination vertex id>:<uuid>, keep in the payload the
message.

the db could be destroyed and created at each superstep by each worker.

as far as combiners are concerned, they could be called (one the second endpoint) after each
get() from the db at compute() time. We're trying to save memory space, not disk. They would
be processed before each vertex's compute() and you'd be required to have all the messages
sent to that vertex  (in that superstep) in memory, because combiners' can't work incrementally.

what do you guys think?
                
> Improve the way to keep outgoing messages
> -----------------------------------------
>
>                 Key: GIRAPH-45
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-45
>             Project: Giraph
>          Issue Type: Improvement
>          Components: bsp
>            Reporter: Hyunsik Choi
>            Assignee: Hyunsik Choi
>
> As discussed in GIRAPH-12(http://goo.gl/CE32U), I think that there is a potential problem
to cause out of memory when the rate of message generation is higher than the rate of message
flush (or network bandwidth).
> To overcome this problem, we need more eager strategy for message flushing or some approach
to spill messages into disk.
> The below link is Dmitriy's suggestion.
> https://issues.apache.org/jira/browse/GIRAPH-12?focusedCommentId=13116253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13116253

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message