spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pacuna <pac...@pm.me>
Subject Spark on K8S - --packages not working for cluster mode?
Date Wed, 05 Jun 2019 20:18:28 GMT
I'm trying to run a sample code that reads a file from s3 so I need the aws
sdk and aws hadoop dependencies.
If I assemble these deps into the main jar everything works fine. But when I
try using --packages, the deps are not seen by the pods.

This is my submit command:

spark-submit 
--master k8s://https://xx.xx.xx.xx
--class "SimpleApp"
--deploy-mode cluster 
--conf spark.kubernetes.container.image=docker.io/pacuna/spark:0.2 
--conf
spark.kubernetes.authenticate.driver.serviceAccountName=spark-test-user 
--packages
com.amazonaws:aws-java-sdk:1.7.4,org.apache.hadoop:hadoop-aws:2.7.3
--conf spark.hadoop.fs.s3a.access.key=... 
--conf spark.hadoop.fs.s3a.secret.key=...  
https://xxxxx/simple-project_2.11-1.0.jar

And the error I'm getting in the driver pod is:

19/06/05 20:13:50 ERROR SparkContext: Failed to add
file:///home/dev/.ivy2/jars/com.fasterxml.jackson.core_jackson-core-2.2.3.jar
to Spark environment                                        
java.io.FileNotFoundException: Jar
/home/dev/.ivy2/jars/com.fasterxml.jackson.core_jackson-core-2.2.3.jar not
found

I'm getting that error for all the deps jars needed.

Any ideas?

Thanks.




--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message