spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srikanth <srikanth...@gmail.com>
Subject Spark 2.0 with Kafka 0.10 exception
Date Tue, 23 Aug 2016 20:44:17 GMT
Hello,

I'm getting the below exception when testing Spark 2.0 with Kafka 0.10.

16/08/23 16:31:01 INFO AppInfoParser: Kafka version : 0.10.0.0
> 16/08/23 16:31:01 INFO AppInfoParser: Kafka commitId : b8642491e78c5a13
> 16/08/23 16:31:01 INFO CachedKafkaConsumer: Initial fetch for
> spark-executor-example mt_event 0 15782114
> 16/08/23 16:31:01 INFO AbstractCoordinator: Discovered coordinator
> 10.150.254.161:9233 (id: 2147483646 rack: null) for group
> spark-executor-example.
> 16/08/23 16:31:02 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID
> 6)
> java.lang.AssertionError: assertion failed: Failed to get records for
> spark-executor-example mt_event 0 15782114 after polling for 512
> at scala.Predef$.assert(Predef.scala:170)
> at
> org.apache.spark.streaming.kafka010.CachedKafkaConsumer.get(CachedKafkaConsumer.scala:74)
> at
> org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:227)
> at
> org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:193)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
>

I get this error intermittently. Sometimes a few batches are scheduled and
run fine. Then I get this error.
kafkacat is able to fetch from this topic continuously.

Full exception is here --
https://gist.github.com/SrikanthTati/c2e95c4ac689cd49aab817e24ec42767

Srikanth

Mime
View raw message