spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gabor Somogyi <gabor.g.somo...@gmail.com>
Subject Re: Data source v2 streaming sinks does not support Update mode
Date Wed, 13 Jan 2021 10:26:59 GMT
Just reached this thread. +1 on to create a simple reproducer app and I
suggest to create a jira attaching the full driver and executor logs.
Ping me on the jira and I'll pick this up right away...

Thanks!

G


On Wed, Jan 13, 2021 at 8:54 AM Jungtaek Lim <kabhwan.opensource@gmail.com>
wrote:

> Would you mind if I ask for a simple reproducer? Would be nice if you
> could create a repository in Github and push the code including the build
> script.
>
> Thanks in advance!
>
> On Wed, Jan 13, 2021 at 3:46 PM Eric Beabes <mailinglists19@gmail.com>
> wrote:
>
>> I tried both. First tried 3.0.0. That didn't work so I tried 3.1.0.
>>
>> On Wed, Jan 13, 2021 at 11:35 AM Jungtaek Lim <
>> kabhwan.opensource@gmail.com> wrote:
>>
>>> Which exact Spark version did you use? Did you make sure the version for
>>> Spark and the version for spark-sql-kafka artifact are the same? (I asked
>>> this because you've said you've used Spark 3.0 but spark-sql-kafka
>>> dependency pointed to 3.1.0.)
>>>
>>> On Tue, Jan 12, 2021 at 11:04 PM Eric Beabes <mailinglists19@gmail.com>
>>> wrote:
>>>
>>>> org.apache.spark.sql.streaming.StreamingQueryException: Data source v2
>>>> streaming sinks does not support Update mode. === Streaming Query ===
>>>> Identifier: [id = 1f342043-29de-4381-bc48-1c6eef99232e, runId =
>>>> 62410f05-db59-4026-83fc-439a79b01c29] Current Committed Offsets: {} Current
>>>> Available Offsets: {} Current State: INITIALIZING Thread State: RUNNABLE
at
>>>> org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:353)
>>>> at
>>>> org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:244)
>>>> Caused by: java.lang.IllegalArgumentException: Data source v2 streaming
>>>> sinks does not support Update mode. at
>>>> org.apache.spark.sql.execution.streaming.StreamExecution.createStreamingWrite(StreamExecution.scala:635)
>>>> at
>>>> org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan$lzycompute(MicroBatchExecution.scala:130)
>>>> at
>>>> org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan(MicroBatchExecution.scala:61)
>>>> at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:320)
>>>> ... 1 more
>>>>
>>>>
>>>> *Please see the attached image for more information.*
>>>>
>>>>
>>>> On Tue, Jan 12, 2021 at 6:01 PM Jacek Laskowski <jacek@japila.pl>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Can you post the whole message? I'm trying to find what might be
>>>>> causing it. A small reproducible example would be of help too. Thank
you.
>>>>>
>>>>> Pozdrawiam,
>>>>> Jacek Laskowski
>>>>> ----
>>>>> https://about.me/JacekLaskowski
>>>>> "The Internals Of" Online Books <https://books.japila.pl/>
>>>>> Follow me on https://twitter.com/jaceklaskowski
>>>>>
>>>>> <https://twitter.com/jaceklaskowski>
>>>>>
>>>>>
>>>>> On Tue, Jan 12, 2021 at 6:35 AM Eric Beabes <mailinglists19@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Trying to port my Spark 2.4 based (Structured) streaming application
>>>>>> to Spark 3.0. I compiled it using the dependency given below:
>>>>>>
>>>>>> <dependency>
>>>>>>     <groupId>org.apache.spark</groupId>
>>>>>>     <artifactId>spark-sql-kafka-0-10_${scala.binary.version}</artifactId>
>>>>>>     <version>3.1.0</version>
>>>>>> </dependency>
>>>>>>
>>>>>>
>>>>>> Every time I run it under Spark 3.0, I get this message: *Data
>>>>>> source v2 streaming sinks does not support Update mode*
>>>>>>
>>>>>> I am using '*mapGroupsWithState*' so as per this link (
>>>>>> https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#output-modes),
>>>>>> the only supported Output mode is "*Update*".
>>>>>>
>>>>>> My Sink is a Kafka topic so I am using this:
>>>>>>
>>>>>> .writeStream
>>>>>> .format("kafka")
>>>>>>
>>>>>>
>>>>>> What am I missing?
>>>>>>
>>>>>>
>>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>
>>>

Mime
View raw message