spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robin East <robin.e...@xense.co.uk>
Subject Re: GraphX path traversal
Date Wed, 04 Mar 2015 16:59:29 GMT
Actually your Pregel code works for me:

import org.apache.spark._
import org.apache.spark.graphx._
import org.apache.spark.rdd.RDD

val vertexlist = Array((1L,"One"), (2L,"Two"), (3L,"Three"), (4L,"Four"),(5L,"Five"),(6L,"Six"))
val edgelist = Array(Edge(6,5,"6 to 5"),Edge(5,4,"5 to 4"),Edge(4,3,"4 to 3"), Edge(3,2,"3
to 2"), Edge(2,1,"2 to 1"))
val vertices: RDD[(VertexId, String)] =  sc.parallelize(vertexlist)
val edges = sc.parallelize(edgelist)
val graph = Graph(vertices, edges)


val parentGraph = Pregel(
      graph.mapVertices((id, attr) => Set[VertexId]()),
      Set[VertexId](),
      Int.MaxValue,
      EdgeDirection.Out)(
        (id, attr, msg) => (msg ++ attr),
        edge => { if (edge.srcId != edge.dstId) 
                  { Iterator((edge.dstId, (edge.srcAttr + edge.srcId))) 
                  } 
                  else Iterator.empty 
                 },
        (a, b) => (a ++ b))
    parentGraph.vertices.collect.foreach(println(_))

Output:

(4,Set(6, 5))
(1,Set(5, 6, 2, 3, 4))
(5,Set(6))
(6,Set())
(2,Set(6, 5, 4, 3))
(3,Set(6, 5, 4))

Maybe your data.csv has edges the wrong way round

Robin

> On 3 Mar 2015, at 16:32, Madabhattula Rajesh Kumar <mrajaforu@gmail.com> wrote:
> 
> Hi,
> 
> I have tried below program using pergel API but I'm not able to get my required output.
I'm getting exactly reverse output which I'm expecting. 
> 
> // Creating graph using above mail mentioned edgefile
>  val graph: Graph[Int, Int] = GraphLoader.edgeListFile(sc, "/home/rajesh/Downloads/graphdata/data.csv").cache()
> 
>  val parentGraph = Pregel(
>       graph.mapVertices((id, attr) => Set[VertexId]()),
>       Set[VertexId](),
>       Int.MaxValue,
>       EdgeDirection.Out)(
>         (id, attr, msg) => (msg ++ attr),
>         edge => { if (edge.srcId != edge.dstId) 
>                   { Iterator((edge.dstId, (edge.srcAttr + edge.srcId))) 
>                   } 
>                   else Iterator.empty 
>                  },
>         (a, b) => (a ++ b))
>     parentGraph.vertices.collect.foreach(println(_))
> 
> Output :
> 
> (4,Set(1, 2, 3))
> (1,Set())
> (6,Set(5, 1, 2, 3, 4))
> (3,Set(1, 2))
> (5,Set(1, 2, 3, 4))
> (2,Set(1))
> 
> But I'm looking below output. 
> 
> (4,Set(5, 6))
> (1,Set(2, 3, 4, 5, 6))
> (6,Set())
> (3,Set(4, 5, 6))
> (5,Set(6))
> (2,Set(3, 4, 5, 6))
> 
> Could you please correct me where I'm doing wrong.
> 
> Regards,
> Rajesh
>  
> 
> On Tue, Mar 3, 2015 at 8:42 PM, Madabhattula Rajesh Kumar <mrajaforu@gmail.com <mailto:mrajaforu@gmail.com>>
wrote:
> Hi Robin,
> 
> Thank you for your response. Please find below my question. I have a below edge file
> 
> Source Vertex	Destination Vertex
> 1	2
> 2	3
> 3	4
> 4	5
> 5	6
> 6	6
> 
> In this graph 1st vertex is connected to 2nd vertex, 2nd Vertex is connected to 3rd vertex,.....
6th vertex is connected to 6th vertex. So 6th vertex is a root node. Please find below graph
> 
> <image.png>
> In this graph, How can I compute the 1st vertex parents like 2,3,4,5,6. Similarly 2nd
vertex parents like 3,4,5,6 .... 6th vertex parent like 6 because this is the root node.
> 
> I'm planning to use pergel API but I'm not able to define messages and vertex program
in that API. Could you please help me on this.
> 
> Please let me know if you need more information.
> 
> Regards,
> Rajesh
> 
> 
> On Tue, Mar 3, 2015 at 8:15 PM, Robin East <robin.east@xense.co.uk <mailto:robin.east@xense.co.uk>>
wrote:
> Rajesh
> 
> I'm not sure if I can help you, however I don't even understand the question. Could you
restate what you are trying to do.
> 
> Sent from my iPhone
> 
> On 2 Mar 2015, at 11:17, Madabhattula Rajesh Kumar <mrajaforu@gmail.com <mailto:mrajaforu@gmail.com>>
wrote:
> 
>> Hi,
>> 
>> I have a below edge list. How to find the parents path for every vertex?
>> 
>> Example :
>> 
>> Vertex 1 path : 2, 3, 4, 5, 6
>> Vertex 2 path : 3, 4, 5, 6
>> Vertex 3 path : 4,5,6
>> vertex 4 path : 5,6
>> vertex 5 path : 6
>> 
>> Could you please let me know how to do this? (or) Any suggestion
>> 
>> Source Vertex	Destination Vertex
>> 1	2
>> 2	3
>> 3	4
>> 4	5
>> 5	6
>> 
>> Regards,
>> Rajesh
> 
> 


Mime
View raw message