spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lisendong <>
Subject union eatch streaming window into a static rdd and use the static rdd periodicity
Date Wed, 06 May 2015 12:50:36 GMT
the pseudo code :

object myApp {
  var myStaticRDD: RDD[Int]
  def main() {
  ...  //init streaming context, and get two DStream (streamA and streamB)
from two hdfs path

  //complex transformation using the two DStream
  val new_stream = streamA.transformWith(StreamB, (a, b, t) => {
  //join the new_stream's rdd with myStaticRDD
  new_stream.foreachRDD(rdd =>
    myStaticRDD = myStaticRDD.join(cur_stream)

  // do complex model training every two hours.
  if (hour is 0, 2, 4, 6...) {
     model_training(myStaticRDD)   //will take 1 hour

I don't know how to write the code to realize training model every two hours
using that moment's myStaticRDD.
And when the model-training is running, the streaming task could also run
normally simultaneously...

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message