spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "L. C. Hsieh (Jira)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-34321) Fix the guarantee of foreachBatch
Date Tue, 02 Feb 2021 05:45:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-34321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

L. C. Hsieh resolved SPARK-34321.
---------------------------------
    Resolution: Invalid

> Fix the guarantee of foreachBatch
> ---------------------------------
>
>                 Key: SPARK-34321
>                 URL: https://issues.apache.org/jira/browse/SPARK-34321
>             Project: Spark
>          Issue Type: Documentation
>          Components: Structured Streaming
>    Affects Versions: 3.2.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Major
>
> Similar to SPARK-28650, {{foreachBatch}} API document also documents the guarantee:
> The batchId can be used to deduplicate and transactionally write the output (that is,
the provided Dataset) to external systems. The output Dataset is guaranteed to be exactly
the same for the same batchId
> But like the reason of fixing the document of {{ForeachWriter}} in SPARK-28650, it is
not hard to break the guarantee by changing the partition number.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message