giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: [jira] [Updated] (GIRAPH-461) Convert static assignment of in-memory partitions with LRU cache
Date Mon, 04 Feb 2013 21:26:51 GMT
The size of the data structure is more or less as big as the previous
out-of-core graph class, and not much bigger than the SimplePartitionStore
(we have a bunch of State enums and Condition objects in two HashMaps).
Anyway the data size is proportional to the number of partitions per
worker, which was 50 with the tested graph (and 63 workers). I'm not so
concerned about memory here.

As far as the number of compute threads is concerned, I would not consider
this test so important. I had 1 worker per core, hence doubling the compute
threads did not buy me much. Plus, a lot of time was spent in the
communication phase at the end, which i did not fine tune.

To really test the approach, I should have 1 worker per machine, and ~1
compute thread per core. I'm going to do it soon :)

Thanks!


On Mon, Feb 4, 2013 at 10:15 PM, Eli Reisman <apache.mailbox@gmail.com>wrote:

> I'll put up a more detailed review soon, but I'm basically +1 on this. I am
> a bit concerned at the many new data structures to be maintained in-memory
> per partition  on each worker task (if I am reading the diff right) to
> maintain the cache. I am happy to see that with the LRU turned on, the 2
> compute thread jobs are slightly faster. If these numbers are correct,
> Claudio has also shown us that the 2 compute thread in-memory job takes
> longer than the single threaded version! I'm hoping these tests were on EC2
> :)
>
> Thanks!
>
> On Sun, Feb 3, 2013 at 6:54 AM, Claudio Martella (JIRA) <jira@apache.org
> >wrote:
>
> >
> >      [
> >
> https://issues.apache.org/jira/browse/GIRAPH-461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
> >
> > Claudio Martella updated GIRAPH-461:
> > ------------------------------------
> >
> >     Attachment: GIRAPH-461.patch
> >
> > Fixed a missing line.
> >
> > > Convert static assignment of in-memory partitions with LRU cache
> > > ----------------------------------------------------------------
> > >
> > >                 Key: GIRAPH-461
> > >                 URL: https://issues.apache.org/jira/browse/GIRAPH-461
> > >             Project: Giraph
> > >          Issue Type: Sub-task
> > >          Components: graph
> > >            Reporter: Claudio Martella
> > >         Attachments: GIRAPH-461.patch, GIRAPH-461.patch,
> GIRAPH-461.patch
> > >
> > >
> > > Currently, the out-of-core partitions are assigned to memory or to disk
> > statically. Using an LRU cache should help keeping in-memory only the
> > partitions that are actively accessed, given a job that does not access
> all
> > the graph at each superstep (traversals) and a good data partitioning
> (non
> > random).
> >
> > --
> > This message is automatically generated by JIRA.
> > If you think it was sent incorrectly, please contact your JIRA
> > administrators
> > For more information on JIRA, see:
> http://www.atlassian.com/software/jira
> >
>



-- 
   Claudio Martella
   claudio.martella@gmail.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message