spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Beabes <mailinglist...@gmail.com>
Subject Re: Data source v2 streaming sinks does not support Update mode
Date Tue, 12 Jan 2021 14:04:13 GMT
org.apache.spark.sql.streaming.StreamingQueryException: Data source v2
streaming sinks does not support Update mode. === Streaming Query ===
Identifier: [id = 1f342043-29de-4381-bc48-1c6eef99232e, runId =
62410f05-db59-4026-83fc-439a79b01c29] Current Committed Offsets: {} Current
Available Offsets: {} Current State: INITIALIZING Thread State: RUNNABLE at
org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:353)
at
org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:244)
Caused by: java.lang.IllegalArgumentException: Data source v2 streaming
sinks does not support Update mode. at
org.apache.spark.sql.execution.streaming.StreamExecution.createStreamingWrite(StreamExecution.scala:635)
at
org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan$lzycompute(MicroBatchExecution.scala:130)
at
org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan(MicroBatchExecution.scala:61)
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:320)
... 1 more


*Please see the attached image for more information.*


On Tue, Jan 12, 2021 at 6:01 PM Jacek Laskowski <jacek@japila.pl> wrote:

> Hi,
>
> Can you post the whole message? I'm trying to find what might be causing
> it. A small reproducible example would be of help too. Thank you.
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books <https://books.japila.pl/>
> Follow me on https://twitter.com/jaceklaskowski
>
> <https://twitter.com/jaceklaskowski>
>
>
> On Tue, Jan 12, 2021 at 6:35 AM Eric Beabes <mailinglists19@gmail.com>
> wrote:
>
>> Trying to port my Spark 2.4 based (Structured) streaming application to
>> Spark 3.0. I compiled it using the dependency given below:
>>
>> <dependency>
>>     <groupId>org.apache.spark</groupId>
>>     <artifactId>spark-sql-kafka-0-10_${scala.binary.version}</artifactId>
>>     <version>3.1.0</version>
>> </dependency>
>>
>>
>> Every time I run it under Spark 3.0, I get this message: *Data source v2
>> streaming sinks does not support Update mode*
>>
>> I am using '*mapGroupsWithState*' so as per this link (
>> https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#output-modes),
>> the only supported Output mode is "*Update*".
>>
>> My Sink is a Kafka topic so I am using this:
>>
>> .writeStream
>> .format("kafka")
>>
>>
>> What am I missing?
>>
>>
>>

Mime
View raw message