spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: Queries with streaming sources must be executed with writeStream.start()
Date Sat, 09 Sep 2017 23:40:37 GMT
yes it is a streaming dataset. so what is the problem with following code?

Dataset<String> ds = dataset.toJSON().map(()->{some function that
returns a string});
 StreamingQuery query = ds.writeStream().start();
 query.awaitTermination();


On Sat, Sep 9, 2017 at 4:20 PM, Felix Cheung <felixcheung_m@hotmail.com>
wrote:

> What is newDS?
> If it is a Streaming Dataset/DataFrame (since you have writeStream there)
> then there seems to be an issue preventing toJSON to work.
>
> ------------------------------
> *From:* kant kodali <kanth909@gmail.com>
> *Sent:* Saturday, September 9, 2017 4:04:33 PM
> *To:* user @spark
> *Subject:* Queries with streaming sources must be executed with
> writeStream.start()
>
> Hi All,
>
> I  have the following code and I am not sure what's wrong with it? I
> cannot call dataset.toJSON() (which returns a DataSet) ? I am using spark
> 2.2.0 so I am wondering if there is any work around?
>
>  Dataset<String> ds = newDS.toJSON().map(()->{some function that returns a string});
>  StreamingQuery query = ds.writeStream().start();
>  query.awaitTermination();
>
>

Mime
View raw message