spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix Cheung <felixcheun...@hotmail.com>
Subject Re: Queries with streaming sources must be executed with writeStream.start()
Date Sat, 09 Sep 2017 23:20:03 GMT
What is newDS?
If it is a Streaming Dataset/DataFrame (since you have writeStream there) then there seems
to be an issue preventing toJSON to work.

________________________________
From: kant kodali <kanth909@gmail.com>
Sent: Saturday, September 9, 2017 4:04:33 PM
To: user @spark
Subject: Queries with streaming sources must be executed with writeStream.start()

Hi All,

I  have the following code and I am not sure what's wrong with it? I cannot call dataset.toJSON()
(which returns a DataSet) ? I am using spark 2.2.0 so I am wondering if there is any work
around?


 Dataset<String> ds = newDS.toJSON().map(()->{some function that returns a string});
 StreamingQuery query = ds.writeStream().start();
 query.awaitTermination();

Mime
View raw message