spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lalit1303 <la...@sigmoidanalytics.com>
Subject Re: Manipulating RDDs within a DStream
Date Fri, 31 Oct 2014 06:48:19 GMT
Hi,

Since, the cassandra object is not serializable you can't open the
connection on driver level and access the object inside foreachRDD (i.e. at
worker level).
You have to open connection inside foreachRDD only, perform the operation
and then close the connection.

For example:

 wordCounts.foreachRDD( rdd => {

       val arr = rdd.toArray
       
       OPEN cassandra connection
       store arr 
       CLOSE cassandra connection

})


Thanks



-----
Lalit Yadav
lalit@sigmoidanalytics.com
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Manipulating-RDDs-within-a-DStream-tp17740p17800.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message