spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rajat kumar <kumar.rajat20...@gmail.com>
Subject Process each kafka record for structured streaming
Date Wed, 20 Jan 2021 09:43:56 GMT
Hi,

I want to apply custom logic for each row of data I am getting through
kafka and want to do it with microbatch.
When I am running it , it is not progressing.


kafka_stream_df \
        .writeStream \
        .foreach(process_records) \
        .outputMode("append") \
        .option("checkpointLocation", "checkpt") \
        .trigger(continuous="5 seconds").start()

Regards

Rajat

Mime
View raw message