spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <>
Subject Re: Spark Streaming : Could not compute split, block not found
Date Fri, 01 Aug 2014 22:58:52 GMT
I meant are you using RDD generated by DStreams, in Spark jobs out
side the DStreams computation?
Something like this:

var globalRDD = null

dstream.foreachRDD(rdd =>
   // have a global pointer based on the rdds generate by dstream
    if (runningFirstTime) globalRDD = rdd

// much much time later try to use the RDD in Spark jobs independent
of the streaming computation

On Fri, Aug 1, 2014 at 3:52 PM, Kanwaldeep <> wrote:
> All the operations being done are using the dstream. I do read an RDD in
> memory which is collected and converted into a map and used for lookups as
> part of DStream operations. This RDD is loaded only once and converted into
> map that is then used on streamed data.
> Do you mean non streaming jobs on RDD using raw kafka data?
> Log File attached:
> streaming.gz
> <>
> --
> View this message in context:
> Sent from the Apache Spark User List mailing list archive at

View raw message