spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Mistroni <>
Subject Re: Run spark-shell inside Docker container against remote YARN cluster
Date Thu, 27 Oct 2016 13:38:06 GMT
I am running spark inside docker though not connecting to cluster
How did u build spark? Which profile did u use?
Pls share details and I can try to replicate

On 27 Oct 2016 2:30 pm, "ponkin" <> wrote:

May be someone already had experience to build docker image for spark?
I want to build docker image with spark inside but configured against remote
YARN cluster.
I have already created image with spark 1.6.2 inside.
But when I run
spark-shell --master yarn --deploy-mode client --driver-memory 32G
--executor-memory 32G --executor-cores 8
inside docker I get the following exception
Diagnostics: File
file:/usr/local/spark/lib/spark-assembly-1.6.2-hadoop2.2.0.jar does not

Any suggestions?
Do I need to load spark-assembly i HDFS and set
spark.yarn.jar=hdfs://spark-assembly-1.6.2-hadoop2.2.0.jar ?

Here is my Dockerfile

View this message in context: http://apache-spark-user-list.
Sent from the Apache Spark User List mailing list archive at

To unsubscribe e-mail:

View raw message