spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From German Schiavon <gschiavonsp...@gmail.com>
Subject Re: Missing required configuration "partition.assignment.strategy" [ Kafka + Spark Structured Streaming ]
Date Mon, 07 Dec 2020 08:29:31 GMT
Hi,

I think the issue is that you are overriding the kafka-clients that comes
with  <artifactId>spark-sql-kafka-0-10_2.12</artifactId>


I'd try removing the kafka-clients and see if it works


On Sun, 6 Dec 2020 at 08:01, Amit Joshi <mailtojoshiamit@gmail.com> wrote:

> Hi All,
>
> I am running the Spark Structured Streaming along with Kafka.
> Below is the pom.xml
>
> <properties>
>     <maven.compiler.source>1.8</maven.compiler.source>
>     <maven.compiler.target>1.8</maven.compiler.target>
>     <encoding>UTF-8</encoding>
>     <!-- Put the Scala version of the cluster -->
>     <scalaVersion>2.12.10</scalaVersion>
>     <sparkVersion>3.0.1</sparkVersion>
> </properties>
>
> <dependency>
>     <groupId>org.apache.kafka</groupId>
>     <artifactId>kafka-clients</artifactId>
>     <version>2.1.0</version>
> </dependency>
>
> <dependency>
>     <groupId>org.apache.spark</groupId>
>     <artifactId>spark-core_2.12</artifactId>
>     <version>${sparkVersion}</version>
>     <scope>provided</scope>
> </dependency>
> <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
> <dependency>
>     <groupId>org.apache.spark</groupId>
>     <artifactId>spark-sql_2.12</artifactId>
>     <version>${sparkVersion}</version>
>     <scope>provided</scope>
> </dependency>
> <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql-kafka-0-10 -->
> <dependency>
>     <groupId>org.apache.spark</groupId>
>     <artifactId>spark-sql-kafka-0-10_2.12</artifactId>
>     <version>${sparkVersion}</version>
> </dependency>
>
> Building the fat jar with shade plugin. The jar is running as expected in my local setup
with the command
>
> *spark-submit --master local[*] --class com.stream.Main --num-executors 3 --driver-memory
2g --executor-cores 2 --executor-memory 3g prism-event-synch-rta.jar*
>
> But when I am trying to run same jar in spark cluster using yarn with command:
>
> *spark-submit --master yarn --deploy-mode cluster --class com.stream.Main --num-executors
4 --driver-memory 2g --executor-cores 1 --executor-memory 4g  gs://jars/prism-event-synch-rta.jar*
>
> Getting the this exception:
>
> 	
>
>
> *at org.apache.spark.sql.execution.streaming.StreamExecution.org <http://org.apache.spark.sql.execution.streaming.StreamExecution.org>$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:355)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:245)Caused
by: org.apache.kafka.common.config.ConfigException: Missing required configuration "partition.assignment.strategy"
which has no default value.	at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:124)*
>
> I have tried setting up the "partition.assignment.strategy", then also its not working.
>
> Please help.
>
>
> Regards
>
> Amit Joshi
>
>

Mime
View raw message