giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dionysios Logothetis (Jira)" <j...@apache.org>
Subject [jira] [Resolved] (GIRAPH-1190) Improve memory usage of SendWorkerOneMessageToManyRequest
Date Fri, 08 May 2020 01:11:00 GMT

     [ https://issues.apache.org/jira/browse/GIRAPH-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dionysios Logothetis resolved GIRAPH-1190.
------------------------------------------
    Resolution: Won't Fix

> Improve memory usage of SendWorkerOneMessageToManyRequest 
> ----------------------------------------------------------
>
>                 Key: GIRAPH-1190
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-1190
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Dionysios Logothetis
>            Assignee: Dionysios Logothetis
>            Priority: Major
>
> The current implementation takes incoming messages stored as ByteArrayOneMessageToManyIds
and prepares them as a map from partition id to a ByteArrayVertexIdMessages, which holds the
messages for the corresponding partition. It then adds these to the message store.
> However, it is possible that these intermediate lists of message get big before they
are added to the message store. If they reach the capacity of the underlying buffers, the
job fails. This can be avoided if we push these lists to the message store before the get
big. This is mostly beneficial when we use a combiner in which case the message store keeps
only one value per vertex.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message