spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: Process each kafka record for structured streaming
Date Thu, 21 Jan 2021 11:41:29 GMT
Hi,

Can you use console sink and make sure that the pipeline shows some
progress?

Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
"The Internals Of" Online Books <https://books.japila.pl/>
Follow me on https://twitter.com/jaceklaskowski

<https://twitter.com/jaceklaskowski>


On Wed, Jan 20, 2021 at 10:44 AM rajat kumar <kumar.rajat20del@gmail.com>
wrote:

> Hi,
>
> I want to apply custom logic for each row of data I am getting through
> kafka and want to do it with microbatch.
> When I am running it , it is not progressing.
>
>
> kafka_stream_df \
>         .writeStream \
>         .foreach(process_records) \
>         .outputMode("append") \
>         .option("checkpointLocation", "checkpt") \
>         .trigger(continuous="5 seconds").start()
>
> Regards
>
> Rajat
>
>
>

Mime
View raw message