spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Quentin Ambard (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-24720) kafka transaction creates Non-consecutive Offsets (due to transaction offset) making streaming fail when failOnDataLoss=true
Date Thu, 02 Aug 2018 20:28:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-24720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Quentin Ambard updated SPARK-24720:
-----------------------------------
    Component/s:     (was: Structured Streaming)
                 DStreams

> kafka transaction creates Non-consecutive Offsets (due to transaction offset) making
streaming fail when failOnDataLoss=true
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-24720
>                 URL: https://issues.apache.org/jira/browse/SPARK-24720
>             Project: Spark
>          Issue Type: Bug
>          Components: DStreams
>    Affects Versions: 2.3.1
>            Reporter: Quentin Ambard
>            Priority: Major
>
> When kafka transactions are used, sending 1 message to kafka will result to 1 offset
for the data + 1 offset to mark the transaction.
> When kafka connector for spark streaming read a topic with non-consecutive offset, it
leads to a failure. SPARK-17147 fixed this issue for compacted topics.
>  However, SPARK-17147 doesn't fix this issue for kafka transactions: if 1 message +
1 transaction commit are in a partition, spark will try to read offsets  [0 2[. offset 0
(containing the message) will be read, but offset 1 won't return a value and buffer.hasNext()
will be false even after a poll since no data are present for offset 1 (it's the transaction
commit)
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message