spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cody Koeninger <c...@koeninger.org>
Subject Re: [Structured Streaming] Kafka group.id is fixed
Date Fri, 09 Nov 2018 17:31:44 GMT
That sounds reasonable to me
On Fri, Nov 9, 2018 at 2:26 AM Anastasios Zouzias <zouzias@gmail.com> wrote:
>
> Hi all,
>
> I run in the following situation with Spark Structure Streaming (SS) using Kafka.
>
> In a project that I work on, there is already a secured Kafka setup where ops can issue
an SSL certificate per "group.id", which should be predefined (or hopefully its prefix to
be predefined).
>
> On the other hand, Spark SS fixes the group.id to
>
> val uniqueGroupId = s"spark-kafka-source-${UUID.randomUUID}-${metadataPath.hashCode}"
>
> see, i.e.,
>
> https://github.com/apache/spark/blob/v2.4.0/external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala#L124
>
> I guess Spark developers had a good reason to fix it, but is it possible to make configurable
the prefix of the above uniqueGroupId ("spark-kafka-source")? If so, I could prepare a PR
on it.
>
> The rational is that we do not want all spark-jobs to use the same certificate on group-ids
of the form (spark-kafka-source-*).
>
>
> Best regards,
> Anastasios Zouzias

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message