giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alessandro Presta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-515) More efficient and flexible edge-based input
Date Thu, 14 Feb 2013 18:31:13 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578555#comment-13578555
] 

Alessandro Presta commented on GIRAPH-515:
------------------------------------------

(continued from previous)

- ByteArrayEdges: an edge list (for the same source vertex) stored as a byte-array. The standard
way of iterating is by reusing Edge objects, but an alternative iterator that instantiates
new objects is provided. Depending on the vertex implementation, we use one of the other.
This is a refactor of the byte-array code in RepresentativeVertex, which now contains an instance
of ByteArrayEdges.
When calling setEdges(), RepresentativeVertex is smart to realize that the passed Iterable
is actually an instance of ByteArrayEdges, and simply takes ownership of it (without iterating).
If using something like EdgeListVertex (which keeps references to the passed edges), we will
use the alternative iterable (this is of course less memory-efficient).

I've also renamed RepresentativeVertex to ByteArrayVertex because it was misleading (it doesn't
need to be used with ByteArrayPartition, it's perfectly fine to have multiple Vertex objects,
each storing its edges in a byte-array).

Future work:

EdgeStore could become an interface in the future, allowing for different implementations
(e.g. out-of-core) and handling permanent edge storage in place of Vertex. That way, we would
have only one Vertex class, and pluggable storage implementations (which makes it easier to
switch without changing user code).


                
> More efficient and flexible edge-based input
> --------------------------------------------
>
>                 Key: GIRAPH-515
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-515
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Alessandro Presta
>            Assignee: Alessandro Presta
>         Attachments: GIRAPH-515.patch
>
>
> The current implementation of edge-based input using mutations is not as memory-efficient
as it could be, and also can't be used with immutable vertex classes.
> By having an ad-hoc code path for edges sent during input superstep, we can achieve both
memory efficiency and eliminate the current restriction.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message