spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oshi <oshc...@gmail.com>
Subject Re: GraphX: Types for the Nodes and Edges
Date Tue, 07 Oct 2014 12:29:34 GMT
Hi again, 

Thank you for your suggestion :) 
I've tried to implement this method but I'm stuck trying to union the
payload before creating the graph. 
Below is a really simplified snippet of what have worked so far.

//Reading the articles given in json format
val articles = sqlContext.jsonFile(path)

articles.registerTempTable("TblArticles")

//Creating two abstract vertex types
class Vertices()
case class Paper(pid: Long, ptitle:String, aid:String) extends Vertices
case class Author(aid:String, aname:String) extends Vertices

//Using the subclasses as payload  with only the fields I need
val filteredPapers=sqlContext.sql("SELECT id,title,authorid FROM
TblArticles")
val vP=filteredPapers.map(line => new Paper(line.getLong(0),
line.getString(1), line.getString(2)).cache

val filteredAuthors=sqlContext.sql("SELECT authorid, name FROM TblArticles")
val vA=filteredAuthors.map(line => new Author(line.getString(0),
line.getString(1)).cache

//Let's assume for now there's only one edge type - "AUTHORS"
val Authedges: RDD[Edge[String]] = vP.map { t => Edge(t.authorid.toLong,
t.id, "AUTHORS")}

1. vP and vA are RDDs, how do I convert them to vertexRDDs and perform the
union?
2. Should the graph be then created as
    val graph=Graph[Vertices,Authedges,String]

Thanks!




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/GraphX-Types-for-the-Nodes-and-Edges-tp15486p15842.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message