spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shixiong Zhu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-28605) Performance regression in SS's foreach
Date Fri, 02 Aug 2019 18:04:00 GMT
Shixiong Zhu created SPARK-28605:
------------------------------------

             Summary: Performance regression in SS's foreach
                 Key: SPARK-28605
                 URL: https://issues.apache.org/jira/browse/SPARK-28605
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 2.4.3
            Reporter: Shixiong Zhu


When "ForeachWriter.open" return "false", ForeachSink v1 will skip the whole partition without
reading data. But in ForeachSink v2, due to the API limitation, it needs to read the whole
partition even if all data just gets dropped.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message