spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ramaswamy, Muthuraman" <>
Subject PySpark Structured Streaming : Writing to DB in Python and Foreach Sink.
Date Tue, 27 Mar 2018 20:12:46 GMT
Hi All,

I am exploring PySpark Structured Streaming and the documentation says the Foreach Sink is
not supported in Python and is available only with Java/Scala. Given the unavailability of
this sink, what options are there for the following:

  1.  Will there be support for Foreach Sink in Python in future Spark Structured Streaming
  2.  What options are there to write streaming query output to Database?
     *   In other words, the streaming query output should be written to a database at every
trigger interval
     *   I cannot use Memory sink as it is recommended for use only with Debug.

Any suggestions to write to database in PySpark Structured Streaming will help. Appreciate
your time.

Thank you,

Muthu Ramaswamy
View raw message