spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andre Piwoni <>
Subject Semantics of Manual Offset Commit for Kafka Spark Streaming
Date Mon, 14 Oct 2019 23:00:22 GMT
When using manual Kafka offset commit in Spark streaming  job and application fails to process
current batch without committing offset in executor, is it expected behavior that next batch
will be processed and offset will be moved to next batch regardless of application failure
to commit? It seems so based on glance at the code. If so, is it expected that job termination
upon failure to process batch and commit offset should resume from last committed offset?

I’m asking since until now I didn’t have to deal with Spark streaming from Kafka where
assumption was “successfully processed  at-least-once”. Stopping Kafka processing or streaming
on any application failure may seem rather extreme but it is what it is.

Thank you,
View raw message