giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman (JIRA)" <>
Subject [jira] [Updated] (GIRAPH-13) Port Giraph to YARN
Date Sat, 16 Feb 2013 01:45:15 GMT


Eli Reisman updated GIRAPH-13:

    Attachment: GIRAPH-13-2.patch

Coming along really nicely, AppMaster and Client are done, utils are done, Maven profile is
configured and working, only compiles for versions of Hadoop that have the YARN deps I need.
Munge flag added but not used yet, should (if I get lucky) only need this to stitch us into
GiraphRunner (one or two lines) and run the client instead of GiraphJob.

Still need to handle a few issues concerning serializing GiraphConf and getting it to the
AppMaster and then our task containers, and telling the RM we're done with the job, but these
are just easy details.

The only real scary challenge yet will be pondering how to allow YARN to fire up the IO formats
and some of our early worker/master setup without Hadoop handing them a JobContext, TaskAttemptContext
Might have to factor out some interfaces for this to happen without gutting our existing IO
code (which I plan to avoid if I can.)

> Port Giraph to YARN
> -------------------
>                 Key: GIRAPH-13
>                 URL:
>             Project: Giraph
>          Issue Type: New Feature
>            Reporter: Jakob Homan
>            Assignee: Eli Reisman
>         Attachments: GIRAPH-13-1.patch, GIRAPH-13-2.patch
> Now that YARN (aka MR2 aka MAPREDUCE-279) has been merged into the Hadoop trunk, we should
think about what it would take to separate out the graph processing bits of Giraph from the
MR1-specific code so as to take advantage of the less-MR centric aspects of YARN, while still
supporting both over the medium term.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message