spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: Spark Streaming : Could not compute split, block not found
Date Fri, 01 Aug 2014 22:58:52 GMT
I meant are you using RDD generated by DStreams, in Spark jobs out
side the DStreams computation?
Something like this:



var globalRDD = null

dstream.foreachRDD(rdd =>
   // have a global pointer based on the rdds generate by dstream
    if (runningFirstTime) globalRDD = rdd
)
ssc.start()
.....

// much much time later try to use the RDD in Spark jobs independent
of the streaming computation
globalRDD.count()










On Fri, Aug 1, 2014 at 3:52 PM, Kanwaldeep <kanwal239@gmail.com> wrote:
> All the operations being done are using the dstream. I do read an RDD in
> memory which is collected and converted into a map and used for lookups as
> part of DStream operations. This RDD is loaded only once and converted into
> map that is then used on streamed data.
>
> Do you mean non streaming jobs on RDD using raw kafka data?
>
> Log File attached:
> streaming.gz
> <http://apache-spark-user-list.1001560.n3.nabble.com/file/n11229/streaming.gz>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Could-not-compute-split-block-not-found-tp11186p11229.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message