hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From developer wang <developer...@gmail.com>
Subject Re: Some question about hama-0.6.3
Date Thu, 27 Feb 2014 13:06:10 GMT
Hi, all.
  Thank you for your detailed reply.

  In the previous test, I used a incomplete graph to run PageRank, then I
got this error:
  java.lang.IllegalArgumentException: Messages must never be behind the
vertex in ID! Current Message ID:

  With your detailed reply, I knew it was because some vertices tried to
send messages to dangling nodes (Actually 0.6.0 can handle this by adding a
repair phase). Then I fixed this by adding dangling nodes explicitly with a
line which only contains a vertex id.

  After this, I could run the PageRank (in the attachment) with the
ListVerticesInfo.

  But if I use DiskVerticesInfo instead of DiskVerticesInfo by this:
  pageJob.set("hama.graph.vertices.info",
"org.apache.hama.graph.DiskVerticesInfo");

  I will still get the below error:
 java.lang.IllegalArgumentException: Messages must never be behind the
vertex in ID! Current Message ID:

 What's the problem?
 Do I use DiskVerticesInfo correctly?

 Or if I want run my application with fault tolerance, what should I do?

 Thank you very much.



2014-02-26 18:27 GMT+08:00 Edward J. Yoon <edwardyoon@apache.org>:

> > Could you answer this question:
> > I found during the loading,  peers would not exchange vertices with each
> > other as hama 0.6.0 did.
> > So how does hama 0.6.3 solve the problem below: a peer load a vertex
> which
> > is belong to another peer? (for example, suppose 3 peers for this task
> and
> > the partitoner is Hash, peer #1 loads vertex 2, in 0.6.3, peer #2 did not
> > send vertex 2 to peer #2)
>
> Instead of network communication, 0.6.3 uses file communication for
> input data partitioning. Please see
>
> http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/PartitioningRunner.java
>
> On Wed, Feb 26, 2014 at 6:03 PM, developer wang <developer.pw@gmail.com>
> wrote:
> > Actually I comment the set statement
> >     //pageJob.set("hama.graph.self.ref", "true");
> >
> > and In GraphJobRunner:
> >     final boolean selfReference = conf.getBoolean("hama.graph.self.ref",
> > false);
> >
> > And I will explicitly set hama.graph.self.ref to false, and use a
> complete
> > graph to have a try again.
> >
> >
> > Could you answer this question:
> > I found during the loading,  peers would not exchange vertices with each
> > other as hama 0.6.0 did.
> > So how does hama 0.6.3 solve the problem below: a peer load a vertex
> which
> > is belong to another peer? (for example, suppose 3 peers for this task
> and
> > the partitoner is Hash, peer #1 loads vertex 2, in 0.6.3, peer #2 did not
> > send vertex 2 to peer #2)
> >
> > or I have some misunderstanding about hama 0.6.3 or above? (In the last
> > years, I used 0.6.0 to do the daily job)
> >
> >
> >
> > 2014-02-26 16:48 GMT+08:00 Edward J. Yoon <edwardyoon@apache.org>:
> >
> >> The background is described here:
> >> https://issues.apache.org/jira/browse/HAMA-758
> >>
> >> On Wed, Feb 26, 2014 at 5:38 PM, Edward J. Yoon <edwardyoon@apache.org>
> >> wrote:
> >> > Oh, please try after set "hama.check.missing.vertex" to false in job
> >> > configuration.
> >> >
> >> > On Wed, Feb 26, 2014 at 5:14 PM, developer wang <
> developer.pw@gmail.com>
> >> > wrote:
> >> >> Thank you very much.
> >> >>
> >> >> Since I think the framework should not decide whether the graph
> should
> >> >> self-reference, so I disable this config. (Actually when I used
> 0.6.0,
> >> >> I
> >> >> also disabled this config)
> >> >>
> >> >> Since I use my PC to test whether my application works, I use a small
> >> >> graph.
> >> >> (It does have a lot of dangling node)
> >> >>
> >> >> The dataset and the PageRank is attached.
> >> >>
> >> >> Thank you very much.
> >> >>
> >> >>
> >> >> 2014-02-26 16:04 GMT+08:00 Edward J. Yoon <edwardyoon@apache.org>:
> >> >>
> >> >>> Hi Wang,
> >> >>>
> >> >>> Can you send me your input data so that I can debug?
> >> >>>
> >> >>> On Wed, Feb 26, 2014 at 4:55 PM, developer wang
> >> >>> <developer.pw@gmail.com>
> >> >>> wrote:
> >> >>> > Firstly, thank you very much for reply.
> >> >>> >
> >> >>> > But in the log, I found "14/02/25 16:45:00 INFO
> >> >>> > graph.GraphJobRunner:
> >> >>> > 2918
> >> >>> > vertices are loaded into localhost:60340 "
> >> >>> > So it had finished the loading phase. is this true?
> >> >>> >
> >> >>> > Another problem is that:
> >> >>> > I found during the loading,  peers would not exchange vertices
> with
> >> >>> > each
> >> >>> > other as hama 0.6.0 did.
> >> >>> > So how does hama 0.6.3 solve the problem below: a peer load
a
> vertex
> >> >>> > which
> >> >>> > is belong to another peer? (for example, suppose 3 peers for
this
> >> >>> > task
> >> >>> > and
> >> >>> > the partitoner is Hash, peer #1 loads vertex 2, in 0.6.3,
peer #2
> >> >>> > did
> >> >>> > not
> >> >>> > send vertex 2 to peer #2)
> >> >>> >
> >> >>> >
> >> >>> > 2014-02-26 15:46 GMT+08:00 Edward J. Yoon <edwardyoon@apache.org
> >:
> >> >>> >
> >> >>> >> > I tried PageRank with a small input of my own.
> >> >>> >>
> >> >>> >> Hi Wang,
> >> >>> >>
> >> >>> >> This error often occurs when there is a record conversion
error.
> >> >>> >> So,
> >> >>> >> you should check whether the vertex reader works correctly.
> >> >>> >>
> >> >>> >> And, I highly recommend you to use latest TRUNK version[1]
as
> >> >>> >> possible.
> >> >>> >>
> >> >>> >> 1.
> >> >>> >>
> >> >>> >>
> >> >>> >>
> http://wiki.apache.org/hama/GettingStarted#Build_latest_version_from_source
> >> >>> >>
> >> >>> >> Thank you.
> >> >>> >>
> >> >>> >> On Wed, Feb 26, 2014 at 1:44 PM, developer wang
> >> >>> >> <developer.pw@gmail.com>
> >> >>> >> wrote:
> >> >>> >> > Hi, all.
> >> >>> >> >    I am Peng Wang, a student trying to use and learn
Hama.
> >> >>> >> >
> >> >>> >> >    I cloned the develop git repository of Hama.
> >> >>> >> >
> >> >>> >> >    I firstly tried the newest version in the tag,
the tag:
> >> >>> >> > 0.7.0-SNAPSHOT.
> >> >>> >> > commit bef419747695d15de8a1087f44028ee40571b5f9
> >> >>> >> > Author: Edward J. Yoon <edwardyoon@apache.org>
> >> >>> >> > Date:   Fri Mar 29 00:44:59 2013 +0000
> >> >>> >> >
> >> >>> >> >     [maven-release-plugin]  copy for tag 0.7.0-SNAPSHOT
> >> >>> >> >
> >> >>> >> >     git-svn-id:
> >> >>> >> >
> https://svn.apache.org/repos/asf/hama/tags/0.7.0-SNAPSHOT@1462366
> >> >>> >> > 13f79535-47bb-0310-9956-ffa450edef68
> >> >>> >> >
> >> >>> >> >   But the tag: 0.6.3-RC3
> >> >>> >> > commit c9526b1272c83d641332667ce5d81d7ccc94be06
> >> >>> >> > Author: Edward J. Yoon <edwardyoon@apache.org>
> >> >>> >> > Date:   Sun Oct 6 08:27:00 2013 +0000
> >> >>> >> >
> >> >>> >> >     [maven-release-plugin]  copy for tag 0.6.3-RC3
> >> >>> >> >
> >> >>> >> >     git-svn-id:
> >> >>> >> > https://svn.apache.org/repos/asf/hama/tags/0.6.3-RC3@1529594
> >> >>> >> > 13f79535-47bb-0310-9956-ffa450edef68
> >> >>> >> >
> >> >>> >> >
> >> >>> >> >    From the commit log, 0.7.0-SNAPSHOT is earlier
than
> 0.6.3-RC3,
> >> >>> >> >    So I used 0.6.3-RC3 instead of 0.7.0-SNAPSHOT
(but on the
> >> >>> >> > website
> >> >>> >> > of
> >> >>> >> > hama, 0.7.0-SNAPSHOT is the newest version)
> >> >>> >> >
> >> >>> >> >    Then I deployed Hama with the Pseudo Distributed
Mode on my
> >> >>> >> > desktop
> >> >>> >> > with
> >> >>> >> > 3 task runners.
> >> >>> >> >    I tried PageRank with a small input of my own.
> >> >>> >> >    But it failes. And its log is:
> >> >>> >> > java.lang.IllegalArgumentException: Messages must
never be
> behind
> >> >>> >> > the
> >> >>> >> > vertex
> >> >>> >> > in ID! Current Message ID: 100128 vs. 1004
> >> >>> >> >         at
> >> >>> >> >
> >> >>> >> >
> org.apache.hama.graph.GraphJobRunner.iterate(GraphJobRunner.java:306)
> >> >>> >> >         at
> >> >>> >> >
> >> >>> >> >
> >> >>> >> >
> >> >>> >> >
> org.apache.hama.graph.GraphJobRunner.doSuperstep(GraphJobRunner.java:254)
> >> >>> >> >         at
> >> >>> >> >
> org.apache.hama.graph.GraphJobRunner.bsp(GraphJobRunner.java:145)
> >> >>> >> >         at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:177)
> >> >>> >> >         at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
> >> >>> >> >         at
> >> >>> >> >
> >> >>> >> >
> >> >>> >> >
> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
> >> >>> >> >
> >> >>> >> >    Could you tell me what is the problem in my situation?
> >> >>> >> >
> >> >>> >> >    I check whether hama had finished the loading
phase, and I
> >> >>> >> > found
> >> >>> >> > "14/02/25 16:45:00 INFO graph.GraphJobRunner: 2918
vertices are
> >> >>> >> > loaded
> >> >>> >> > into
> >> >>> >> > localhost:60340 "in the log.
> >> >>> >> >    So it had finished the loading phase.
> >> >>> >> >
> >> >>> >> >    After this, I read the source code, and I found
during the
> >> >>> >> > loading,
> >> >>> >> > peers would not exchange vertices with each other
as hama 0.5.0
> >> >>> >> > did.
> >> >>> >> >    So how does hama 0.6.3 solve the problem below:
a peer load
> a
> >> >>> >> > vertex
> >> >>> >> > which is belong to another peer?
> >> >>> >> >
> >> >>> >> >    Could you tell which branch or tag is a stable
version?
> >> >>> >> >    And does it support fault tolerance for graph
algorithms?
>  and
> >> >>> >> > how
> >> >>> >> > can I
> >> >>> >> > get it?
> >> >>> >> >
> >> >>> >> >
> >> >>> >> >
> >> >>> >>
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >> Edward J. Yoon (@eddieyoon)
> >> >>> >> Chief Executive Officer
> >> >>> >> DataSayer, Inc.
> >> >>> >
> >> >>> >
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Edward J. Yoon (@eddieyoon)
> >> >>> Chief Executive Officer
> >> >>> DataSayer, Inc.
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Edward J. Yoon (@eddieyoon)
> >> > Chief Executive Officer
> >> > DataSayer, Inc.
> >>
> >>
> >>
> >> --
> >> Edward J. Yoon (@eddieyoon)
> >> Chief Executive Officer
> >> DataSayer, Inc.
> >
> >
>
>
>
> --
> Edward J. Yoon (@eddieyoon)
> Chief Executive Officer
> DataSayer, Inc.
>

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message