Hi There,

My use case is to read a simple json message from Kafka queue using Spark Structured Streaming. But I’m getting the following error message when I run  my Kafka consumer. I don’t get this error when using Spark direct stream. The issue is happening only with structured streaming. Any help would be greatly appreciated.


Exception in thread "main" org.apache.spark.sql.streaming.StreamingQueryException: Not authorized to access group: spark-kafka-source-060f3ceb-09f4-4e28-8210-3ef8a845fc92--2038748645-driver-2
=== Streaming Query ===
Identifier: [id = 6ab10eab-4f71-435c-8705-820e66cee47e, runId = 48430367-9e14-450b-b8e0-27199b536403]
Current Committed Offsets: {}
Current Available Offsets: {}

 

Current State: ACTIVE
Thread State: RUNNABLE

 

Logical Plan:
KafkaSource[Subscribe[cla-claim-raw]]
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:295)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189)
Caused by: org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: spark-kafka-source-060f3ceb-09f4-4e28-8210-3ef8a845fc92--2038748645-driver-2

Thanks,
Thomas Thomas