spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shuai Zhang <smalldirec...@yahoo.com.INVALID>
Subject The problem when share data inside Dstream
Date Tue, 16 Jun 2015 15:15:43 GMT
Hello guys,
I faced one problem that I cannot pass my data inside rdd partition when I was trying to develop
spark streaming feature.I'm the newcomer of Spark, could you please give me any suggestion
on this problem? 
The figure in the attachment is the code I used in my program:
After I run my code, I found these logs:Inside foreachRDD scope I can get the right indexGeneratorMap
value which includes right kva instance, "CassandraAccessDatastax@5adc7292"
15/06/16 12:07:35 INFO streaming.AbstractSparkStreamingMain: [RDD][Index Generator Map]: {class
com.worksap.company.framework.autoindex.edp.job.SparkStreamingTestEntity=AbstractIndexGenerator{kva=com.worksap.company.access.cassandra.CassandraAccessDatastax@5adc7292,
indexName='spark_streaming_test_index'}}
But in the foreachPartition scope, I found my kva instance is null. I totally have no idea
about it.
15/06/16 12:07:28 INFO streaming.AbstractSparkStreamingMain: [Index Generator Map]: {class
com.worksap.company.framework.autoindex.edp.job.SparkStreamingTestEntity=AbstractIndexGenerator{kva=null,
indexName='spark_streaming_test_index'}}
If someone can give me any tips, I will really appreciate your help. 
Thanks in advance.
Best Regards,
--
Gabriel Zhang
 
Mime
View raw message