spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oshi <>
Subject Re: GraphX: Types for the Nodes and Edges
Date Tue, 07 Oct 2014 12:29:34 GMT
Hi again, 

Thank you for your suggestion :) 
I've tried to implement this method but I'm stuck trying to union the
payload before creating the graph. 
Below is a really simplified snippet of what have worked so far.

//Reading the articles given in json format
val articles = sqlContext.jsonFile(path)


//Creating two abstract vertex types
class Vertices()
case class Paper(pid: Long, ptitle:String, aid:String) extends Vertices
case class Author(aid:String, aname:String) extends Vertices

//Using the subclasses as payload  with only the fields I need
val filteredPapers=sqlContext.sql("SELECT id,title,authorid FROM
val => new Paper(line.getLong(0),
line.getString(1), line.getString(2)).cache

val filteredAuthors=sqlContext.sql("SELECT authorid, name FROM TblArticles")
val => new Author(line.getString(0),

//Let's assume for now there's only one edge type - "AUTHORS"
val Authedges: RDD[Edge[String]] = { t => Edge(t.authorid.toLong,, "AUTHORS")}

1. vP and vA are RDDs, how do I convert them to vertexRDDs and perform the
2. Should the graph be then created as
    val graph=Graph[Vertices,Authedges,String]


View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message