Also check this out

From the data bricks reference app:

From: Ewan Leith
Date: Tuesday, September 29, 2015 at 5:09 PM
To: Daniel Haviv, user
Subject: RE: Converting a DStream to schemaRDD

Something like:


dstream.foreachRDD { rdd =>

  val df =…)




Might be the place to start, it’ll convert each batch of dstream into an RDD then let you work it as if it were a standard RDD dataset.





From: Daniel Haviv []
Sent: 29 September 2015 15:03
To: user <>
Subject: Converting a DStream to schemaRDD



I have a DStream which is a stream of RDD[String].


How can I pass a DStream to sqlContext.jsonRDD and work with it as a DF ?


Thank you.