spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Beabes <mailinglist...@gmail.com>
Subject Re: Spark Structured Streaming: “earliest” as “startingOffsets” is not working
Date Sat, 27 Jun 2020 00:42:22 GMT
My apologies...  After I set the 'maxOffsetsPerTrigger' to a value such as
'200000' it started working. Hopefully this will help someone. Thanks.

On Fri, Jun 26, 2020 at 2:12 PM Something Something <
mailinglists19@gmail.com> wrote:

> My Spark Structured Streaming job works fine when I set "startingOffsets"
> to "latest". When I simply change it to "earliest" & specify a new "check
> point directory", the job doesn't work. The states don't get timed out
> after 10 minutes.
>
> While debugging I noticed that my 'state' logic is indeed getting executed
> but states just don't time out - as they do when I use "latest". Any reason
> why?
>
> Is this a known issue?
>
> *Note*: I've tried this under Spark 2.3 & 2.4
>

Mime
View raw message