spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cody Koeninger <c...@koeninger.org>
Subject Re: "Got wrong record after seeking to offset" issue
Date Thu, 18 Jan 2018 04:12:25 GMT
That means the consumer on the executor tried to seek to the specified
offset, but the message that was returned did not have a matching
offset.  If the executor can't get the messages the driver told it to
get, something's generally wrong.

What happens when you try to consume the particular failing offset
from another  (e.g. commandline) consumer?

Is the topic in question compacted?



On Tue, Jan 16, 2018 at 11:10 PM, Justin Miller
<justin.miller@protectwise.com> wrote:
> Greetings all,
>
> I’ve recently started hitting on the following error in Spark Streaming in Kafka. Adjusting
maxRetries and spark.streaming.kafka.consumer.poll.ms even to five minutes doesn’t seem
to be helping. The problem only manifested in the last few days, restarting with a new consumer
group seems to remedy the issue for a few hours (< retention, which is 12 hours).
>
> Error:
> Caused by: java.lang.AssertionError: assertion failed: Got wrong record for spark-executor-<consumergrouphere>
<topichere> 76 even after seeking to offset 1759148155
>     at scala.Predef$.assert(Predef.scala:170)
>     at org.apache.spark.streaming.kafka010.CachedKafkaConsumer.get(CachedKafkaConsumer.scala:85)
>     at org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:223)
>     at org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:189)
>     at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
>
> I guess my questions are, why is that assertion a job killer vs a warning and is there
anything I can tweak settings wise that may keep it at bay.
>
> I wouldn’t be surprised if this issue were exacerbated by the volume we do on Kafka
topics (~150k/sec on the persister that’s crashing).
>
> Thank you!
> Justin
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message