giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavan Kumar (JIRA)" <>
Subject [jira] [Commented] (GIRAPH-909) support succinct representation of messages in messagestores
Date Fri, 06 Jun 2014 01:51:01 GMT


Pavan Kumar commented on GIRAPH-909:

#duplicate- GIRAPH-912 is the current ipsco-facto

> support succinct representation of messages in messagestores
> ------------------------------------------------------------
>                 Key: GIRAPH-909
>                 URL:
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Pavan Kumar
>            Assignee: Pavan Kumar
> Currently we use ByteArrayVertexIdMessages data-structure to store vertex id & its
messages. Even for ByteArrayOneToAllMessages data-structure, while storing in message-store
we always convert it to ByteArrayVertexIdMessages, in this case if many vertices on a worker
receive the same message, it is stored as many times. This uses up lot of memory, so if we
have message stores that can avoid all this duplication in storing messages, we can bring
down memory footprint by a large amount.
> Note: however, here the pre-req is that your graph has to be partitioned such that a
vertex sends messages to vertices only in a few other workers (not all of them - as happens
with HashPartitioning)
> This change depends heavily on GIRAPH-907 & GIRAPH-908
> I already have patch for it, need to tidy up few things, will put it up by end of this

This message was sent by Atlassian JIRA

View raw message