spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cody Koeninger <c...@koeninger.org>
Subject Re: Spark 2.0 with Kafka 0.10 exception
Date Wed, 24 Aug 2016 01:52:57 GMT
You can set that poll timeout higher with

spark.streaming.kafka.consumer.poll.ms

but half a second is fairly generous.  I'd try to take a look at
what's going on with your network or kafka broker during that time.

On Tue, Aug 23, 2016 at 4:44 PM, Srikanth <srikanth.ht@gmail.com> wrote:
> Hello,
>
> I'm getting the below exception when testing Spark 2.0 with Kafka 0.10.
>
>> 16/08/23 16:31:01 INFO AppInfoParser: Kafka version : 0.10.0.0
>> 16/08/23 16:31:01 INFO AppInfoParser: Kafka commitId : b8642491e78c5a13
>> 16/08/23 16:31:01 INFO CachedKafkaConsumer: Initial fetch for
>> spark-executor-example mt_event 0 15782114
>> 16/08/23 16:31:01 INFO AbstractCoordinator: Discovered coordinator
>> 10.150.254.161:9233 (id: 2147483646 rack: null) for group
>> spark-executor-example.
>> 16/08/23 16:31:02 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID
>> 6)
>> java.lang.AssertionError: assertion failed: Failed to get records for
>> spark-executor-example mt_event 0 15782114 after polling for 512
>> at scala.Predef$.assert(Predef.scala:170)
>> at
>> org.apache.spark.streaming.kafka010.CachedKafkaConsumer.get(CachedKafkaConsumer.scala:74)
>> at
>> org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:227)
>> at
>> org.apache.spark.streaming.kafka010.KafkaRDD$KafkaRDDIterator.next(KafkaRDD.scala:193)
>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
>
>
> I get this error intermittently. Sometimes a few batches are scheduled and
> run fine. Then I get this error.
> kafkacat is able to fetch from this topic continuously.
>
> Full exception is here --
> https://gist.github.com/SrikanthTati/c2e95c4ac689cd49aab817e24ec42767
>
> Srikanth

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message