spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Mistroni <mmistr...@gmail.com>
Subject Re: Run spark-shell inside Docker container against remote YARN cluster
Date Thu, 27 Oct 2016 13:38:06 GMT
I am running spark inside docker though not connecting to cluster
How did u build spark? Which profile did u use?
Pls share details and I can try to replicate
Kr

On 27 Oct 2016 2:30 pm, "ponkin" <alexey.ponkin@ya.ru> wrote:

Hi,
May be someone already had experience to build docker image for spark?
I want to build docker image with spark inside but configured against remote
YARN cluster.
I have already created image with spark 1.6.2 inside.
But when I run
spark-shell --master yarn --deploy-mode client --driver-memory 32G
--executor-memory 32G --executor-cores 8
inside docker I get the following exception
Diagnostics: java.io.FileNotFoundException: File
file:/usr/local/spark/lib/spark-assembly-1.6.2-hadoop2.2.0.jar does not
exist

Any suggestions?
Do I need to load spark-assembly i HDFS and set
spark.yarn.jar=hdfs://spark-assembly-1.6.2-hadoop2.2.0.jar ?

Here is my Dockerfile
https://gist.github.com/ponkin/cac0a071e7fe75ca7c390b7388cf4f91



--
View this message in context: http://apache-spark-user-list.
1001560.n3.nabble.com/Run-spark-shell-inside-Docker-
container-against-remote-YARN-cluster-tp27967.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Mime
View raw message