spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Something Something <mailinglist...@gmail.com>
Subject Spark Structured Streaming: “earliest” as “startingOffsets” is not working
Date Fri, 26 Jun 2020 21:12:04 GMT
My Spark Structured Streaming job works fine when I set "startingOffsets"
to "latest". When I simply change it to "earliest" & specify a new "check
point directory", the job doesn't work. The states don't get timed out
after 10 minutes.

While debugging I noticed that my 'state' logic is indeed getting executed
but states just don't time out - as they do when I use "latest". Any reason
why?

Is this a known issue?

*Note*: I've tried this under Spark 2.3 & 2.4

Mime
View raw message