spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Mocanu <amoc...@verticalscope.com>
Subject sstream.foreachRDD
Date Tue, 04 Mar 2014 16:18:58 GMT
Hi
I've noticed that if in the driver of a spark app I have a foreach and add stream elements
to a list from the stream, the list contains no elements at the end of the processing.

Take this sample code:
  val list= new java.util.List()
  sstream.foreachRDD (rdd => rdd.foreach( tuple => list.add(tuple) ) )

If in the add method of the list I put a print statement I see the tuples added. But when
I print the list it is empty. I do wait for the stream to finish before I print the list so
it's not a timing/racing issue.

I think it might have something to do  with the fact that Spark sends the List code to its
nodes, adds the data to the list there, but never sends the list back to the driver program
or at least in the driver program the pointer to the list does not point do the list that
was over to Spark nodes and which now has the tuples from the stream

Am I supposed to use RDD.collect then add the data to the List or what's the proper way to
get tuples out of Spark?
 sstream.foreachRDD (rdd => rdd.collect.foreach( tuple => list.add(tuple) ) )

Thanks
-Adrian


Mime
View raw message