spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shixiong(Ryan) Zhu" <shixi...@databricks.com>
Subject Re: How to print plan of Structured Streaming DataFrame
Date Mon, 20 Nov 2017 18:57:14 GMT
-dev +user

Which Spark version are you using? There is a bug in the old Spark. Try to
use the latest version.

In addition, you can call `query.explain()` as well.

On Mon, Nov 20, 2017 at 4:00 AM, Chang Chen <baibaichen@gmail.com> wrote:

> Hi Guys
>
> I modified StructuredNetworkWordCount to see what the executed plan is,
> here are my codes:
>
> val wordCounts = words.groupBy("value").count()
>
> // Start running the query that prints the running counts to the console
> val query = wordCounts.writeStream
>   .outputMode("complete")
>   .format("console")
>   .start()
>
> wordCounts.explain()  // additional codes
>
>
> But it failed with “AnalysisException: Queries with streaming sources must
> be executed with writeStream.start()”?
>
>
> Thanks
> Chang
>
>

Mime
View raw message