spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jungtaek Lim <kabhwan.opensou...@gmail.com>
Subject Re: Using startingOffsets latest - no data from structured streaming kafka query
Date Sun, 19 Apr 2020 22:41:48 GMT
Did you provide more records to topic "after" you started the query? That's
the only one I can imagine based on such information.

On Fri, Apr 17, 2020 at 9:13 AM Ruijing Li <liruijing09@gmail.com> wrote:

> Hi all,
>
> Apologies if this has been asked before, but I could not find the answer
> to this question. We have a structured streaming job, but for some reason,
> if we use startingOffsets = latest with foreachbatch mode, it doesn’t
> produce any data.
>
> Rather, in logs I see it repeats the message “ Fetcher [Consumer]
> Resetting offset for partition to offset” over and over again..
>
> However with startingOffsets=earliest, we don’t get this issue. I’m
> wondering then how we can use startingOffsets=latest as I wish to start
> from the latest offset available.
> --
> Cheers,
> Ruijing Li
>

Mime
View raw message