spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gabor Somogyi <gabor.g.somo...@gmail.com>
Subject Re: Exception in thread "main" org.apache.spark.sql.streaming.StreamingQueryException: Not authorized to access group: spark-kafka-source-060f3ceb-09f4-4e28-8210-3ef8a845fc92--2038748645-driver-2
Date Wed, 13 Feb 2019 09:36:19 GMT
Hi Thomas,

The issue occurs when the user does not have the READ permission on the
consumer groups.

In DStreams group ID is configured in application, for example:
https://github.com/gaborgsomogyi/spark-dstream-secure-kafka-app/blob/161bf02eb3677aac604d63499041f72231d0e371/src/main/scala/com/cloudera/spark/examples/DirectKafkaWordCount.scala#L59

In Strucuted Streaming the group ID is generated by Spark internally.

Either one has to give access to "spark-kafka-source-*" group or in Spark
3.0 this prefix can be configured with "groupidprefix" parameter.

BR,
G


On Wed, Feb 13, 2019 at 3:58 AM Allu👌🏽 Thomas
<thomaspt747@icloud.com.invalid> wrote:

> Hi There,
>
> My use case is to read a simple json message from Kafka queue using Spark
> Structured Streaming. But I’m getting the following error message when I
> run  my Kafka consumer. I don’t get this error when using Spark direct
> stream. The issue is happening only with structured streaming. Any help
> would be greatly appreciated.
>
>
> Exception in thread "main"
> org.apache.spark.sql.streaming.StreamingQueryException: Not authorized to
> access group:
> spark-kafka-source-060f3ceb-09f4-4e28-8210-3ef8a845fc92--2038748645-driver-2
> === Streaming Query ===
> Identifier: [id = 6ab10eab-4f71-435c-8705-820e66cee47e, runId =
> 48430367-9e14-450b-b8e0-27199b536403]
> Current Committed Offsets: {}
> Current Available Offsets: {}
>
>
> Current State: ACTIVE
> Thread State: RUNNABLE
>
>
> Logical Plan:
> KafkaSource[Subscribe[cla-claim-raw]]
>         at org.apache.spark.sql.execution.streaming.StreamExecution.org
> $apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:295)
>         at
> org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189)
> Caused by: org.apache.kafka.common.errors.GroupAuthorizationException: Not
> authorized to access group:
> spark-kafka-source-060f3ceb-09f4-4e28-8210-3ef8a845fc92--2038748645-driver-2
>
> Thanks,
> Thomas Thomas
>

Mime
View raw message