spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dick Davies <>
Subject is there a 'knack' to docker and mesos?
Date Sun, 23 Aug 2015 19:20:06 GMT
Really excited to try out the new Docker executor support on 1.4.1, I'm
making progress but feel like I'm missing something.


spark-1.4.1-hadoop2.6 - not using hadoop yet
mac os x yosemite java 8 spark-shell
mesos 0.22.1  : 2 slaves, 1 master + zk , all on centos 6.x
docker 1.8.x

I wanted to use a generic docker image with e.g. Java 8 in it, and then
deploy the 1.4.1 distro into it. The docs seem to indicate that's supported.

Mesos and Docker pull the requested docker images ( 'docker pull java'
and extract the spark.executor.uri distro correctly. It seems to fall
over at cd'ing of
all places:

"cd: can't cd to spark-1*"

this causes the task to fail and eventually have spark blacklist the slave.

Is this just because 'spark-1*' matches both the tarball and the directory?

full mesos stderr looks like:


I0823 19:13:25.608206  3069 fetcher.cpp:214] Fetching URI
I0823 19:13:25.608582  3069 fetcher.cpp:125] Fetching URI
with os::net
I0823 19:13:25.608620  3069 fetcher.cpp:135] Downloading
to '/var/mesos/slaves/20150823-110659-1862270986-5050-3230-S1/frameworks/20150823-191138-1862270986-5050-3768-0000/executors/0/runs/a5a13cd4-013a-4ebc-8ef7-eb9c33358429/spark-1.4.1-bin-hadoop2.6.tgz'
I0823 19:14:37.765060  3069 fetcher.cpp:78] Extracted resource
into '/var/mesos/slaves/20150823-110659-1862270986-5050-3230-S1/frameworks/20150823-191138-1862270986-5050-3768-0000/executors/0/runs/a5a13cd4-013a-4ebc-8ef7-eb9c33358429'
/bin/sh: 1: cd: can't cd to spark-1*
/bin/sh: 1: ./bin/spark-class: not found
I0823 19:14:38.365190  3138 exec.cpp:132] Version: 0.22.1
I0823 19:14:38.369495  3156 exec.cpp:206] Executor registered on slave


and the spark-shell says:

15/08/23 20:15:51 INFO CoarseMesosSchedulerBackend: Mesos task 3 is
15/08/23 20:15:51 INFO CoarseMesosSchedulerBackend: Mesos task 3 is
15/08/23 20:15:51 INFO CoarseMesosSchedulerBackend: Blacklisting Mesos
slave value: "20150823-110659-1862270986-5050-3230-S0"
 due to too many failures; is Spark installed on it?

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message