spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Mocanu <>
Subject RE: StreamingContext.transform on a DStream
Date Thu, 27 Mar 2014 15:59:55 GMT
Please disregard I didn't see the Seq wrapper.

From: Adrian Mocanu []
Sent: March-27-14 11:57 AM
Subject: StreamingContext.transform on a DStream

Found this transform fn in StreamingContext which takes in a DStream[_] and a function which
acts on each of its RDDs
Unfortunately I can't figure out how to transform my DStream[(String,Int)] into DStream[_]

/*** Create a new DStream in which each RDD is generated by applying a function on RDDs of
the DStreams. */
  def transform[T: ClassTag](
      dstreams: Seq[DStream[_]],
      transformFunc: (Seq[RDD[_]], Time) => RDD[T]
    ): DStream[T] = {
    new TransformedDStream[T](dstreams, sparkContext.clean(transformFunc))


View raw message