spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jungtaek Lim <kabhwan.opensou...@gmail.com>
Subject Re: Cannot perform operation after producer has been closed
Date Tue, 03 Nov 2020 04:31:51 GMT
Which Spark version do you use? There's a known issue on Kafka producer
pool in Spark 2.x which was fixed in Spark 3.0, so you'd like to check
whether your case is bound to the known issue or not.

https://issues.apache.org/jira/browse/SPARK-21869


On Tue, Nov 3, 2020 at 1:53 AM Eric Beabes <mailinglists19@gmail.com> wrote:

> I know this is related to Kafka but it happens during the Spark Structured
> Streaming job that's why I am asking on this mailing list.
>
> How would you debug this or get around this in Spark Structured Streaming?
> Any tips would be appreciated. Thanks.
>
>
> java.lang.IllegalStateException: Cannot perform operation after producer
> has been closed at
> org.apache.kafka.clients.producer.KafkaProducer.throwIfProducerClosed(KafkaProducer.java:853)
> at
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:862)
> at
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:846)
> at
> org.apache.spark.sql.kafka010.KafkaRowWriter.sendRow(KafkaWriteTask.scala:92)
> at
> org.apache.spark.sql.kafka010.KafkaStreamDataWriter.write(KafkaStreamWriter.scala:95)
>

Mime
View raw message