spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stavros Kontopoulos <stavros.kontopou...@lightbend.com>
Subject Re: Spark on K8S - --packages not working for cluster mode?
Date Thu, 06 Jun 2019 12:37:37 GMT
Hi,

This has been fixed here: https://github.com/apache/spark/pull/23546. Will
be available with Spark 3.0.0

Best,
Stavros

On Wed, Jun 5, 2019 at 11:18 PM pacuna <pacuna@pm.me> wrote:

> I'm trying to run a sample code that reads a file from s3 so I need the aws
> sdk and aws hadoop dependencies.
> If I assemble these deps into the main jar everything works fine. But when
> I
> try using --packages, the deps are not seen by the pods.
>
> This is my submit command:
>
> spark-submit
> --master k8s://https://xx.xx.xx.xx
> --class "SimpleApp"
> --deploy-mode cluster
> --conf spark.kubernetes.container.image=docker.io/pacuna/spark:0.2
> --conf
> spark.kubernetes.authenticate.driver.serviceAccountName=spark-test-user
> --packages
> com.amazonaws:aws-java-sdk:1.7.4,org.apache.hadoop:hadoop-aws:2.7.3
> --conf spark.hadoop.fs.s3a.access.key=...
> --conf spark.hadoop.fs.s3a.secret.key=...
> https://xxxxx/simple-project_2.11-1.0.jar
>
> And the error I'm getting in the driver pod is:
>
> 19/06/05 20:13:50 ERROR SparkContext: Failed to add
>
> file:///home/dev/.ivy2/jars/com.fasterxml.jackson.core_jackson-core-2.2.3.jar
> to Spark environment
> java.io.FileNotFoundException: Jar
> /home/dev/.ivy2/jars/com.fasterxml.jackson.core_jackson-core-2.2.3.jar not
> found
>
> I'm getting that error for all the deps jars needed.
>
> Any ideas?
>
> Thanks.
>
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message