spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Error while doing stream-stream inner join (java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access)
Date Mon, 02 Jul 2018 10:56:00 GMT
Hi All,

I get the below error quite often when I do an stream-stream inner join on
two data frames. After running through several experiments stream-stream
joins dont look stable enough for production yet. any advice on this?

Thanks!

java.util.ConcurrentModificationException: KafkaConsumer is not safe for
multi-threaded access
18/07/02 09:32:14 INFO LineBufferedStream: stdout:     at
org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1431)
18/07/02 09:32:14 INFO LineBufferedStream: stdout:     at
org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1361)
18/07/02 09:32:14 INFO LineBufferedStream: stdout:     at
org.apache.spark.sql.kafka010.CachedKafkaConsumer.close(CachedKafkaConsumer.scala:301)

Mime
View raw message