spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Tanase <>
Subject Re: Converting a DStream to schemaRDD
Date Tue, 29 Sep 2015 14:13:11 GMT
Also check this out

From the data bricks reference app:

From: Ewan Leith
Date: Tuesday, September 29, 2015 at 5:09 PM
To: Daniel Haviv, user
Subject: RE: Converting a DStream to schemaRDD

Something like:

dstream.foreachRDD { rdd =>
  val df =…)

Might be the place to start, it’ll convert each batch of dstream into an RDD then let you
work it as if it were a standard RDD dataset.


From: Daniel Haviv []
Sent: 29 September 2015 15:03
To: user <<>>
Subject: Converting a DStream to schemaRDD

I have a DStream which is a stream of RDD[String].

How can I pass a DStream to sqlContext.jsonRDD and work with it as a DF ?

Thank you.

View raw message