spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From joe schmoo <joesurf...@gmail.com>
Subject Re: Local Dev Env with Mesos + Spark Streaming on Docker: Can't submit jobs.
Date Thu, 15 May 2014 20:24:33 GMT
I
On May 5, 2014 1:12 PM, "Gerard Maas" <gerard.maas@gmail.com> wrote:
>
> Hi all,
>
> I'm currently working on creating a set of docker images to facilitate
local development with Spark/streaming on Mesos (+zk, hdfs, kafka)
>
> After solving the initial hurdles to get things working together in
docker containers, now everything seems to start-up correctly and the mesos
UI shows slaves as they are started.
>
> I'm trying to submit a job from IntelliJ and the jobs submissions seem to
get lost in Mesos translation. The logs are not helping me to figure out
what's wrong, so I'm posting them here in the hope that they can ring a
bell and somebdoy could provide me a hint on what's wrong/missing with my
setup.
>
>
> ---- DRIVER (IntelliJ running a Job.scala main) ----
> 14/05/05 21:52:31 INFO MetadataCleaner: Ran metadata cleaner for
SHUFFLE_BLOCK_MANAGER
> 14/05/05 21:52:31 INFO BlockManager: Dropping broadcast blocks older than
1399319251962
> 14/05/05 21:52:31 INFO BlockManager: Dropping non broadcast blocks older
than 1399319251962
> 14/05/05 21:52:31 INFO MetadataCleaner: Ran metadata cleaner for
BROADCAST_VARS
> 14/05/05 21:52:31 INFO MetadataCleaner: Ran metadata cleaner for
BLOCK_MANAGER
> 14/05/05 21:52:32 INFO MetadataCleaner: Ran metadata cleaner for
HTTP_BROADCAST
> 14/05/05 21:52:32 INFO MetadataCleaner: Ran metadata cleaner for
MAP_OUTPUT_TRACKER
> 14/05/05 21:52:32 INFO MetadataCleaner: Ran metadata cleaner for
SPARK_CONTEXT
>
>
> ---- MESOS MASTER ----
> I0505 19:52:39.718080   388 master.cpp:690] Registering framework
201405051517-67113388-5050-383-6995 at scheduler(1)@127.0.1.1:58115
> I0505 19:52:39.718261   388 master.cpp:493] Framework
201405051517-67113388-5050-383-6995 disconnected
> I0505 19:52:39.718277   389 hierarchical_allocator_process.hpp:332] Added
framework 201405051517-67113388-5050-383-6995
> I0505 19:52:39.718312   388 master.cpp:520] Giving framework
201405051517-67113388-5050-383-6995 0ns to failover
> I0505 19:52:39.718431   389 hierarchical_allocator_process.hpp:408]
Deactivated framework 201405051517-67113388-5050-383-6995
> W0505 19:52:39.718459   388 master.cpp:1388] Master returning resources
offered to framework 201405051517-67113388-5050-383-6995 because the
framework has terminated or is inactive
> I0505 19:52:39.718567   388 master.cpp:1376] Framework failover timeout,
removing framework 201405051517-67113388-5050-383-6995
>
>
>
> ---- MESOS SLAVE ----
> I0505 19:49:27.662019    20 slave.cpp:1191] Asked to shut down framework
201405051517-67113388-5050-383-6803 by master@172.17.0.4:5050
> W0505 19:49:27.662072    20 slave.cpp:1206] Cannot shut down unknown
framework 201405051517-67113388-5050-383-6803
> I0505 19:49:28.662153    18 slave.cpp:1191] Asked to shut down framework
201405051517-67113388-5050-383-6804 by master@172.17.0.4:5050
> W0505 19:49:28.662212    18 slave.cpp:1206] Cannot shut down unknown
framework 201405051517-67113388-5050-383-6804
> I0505 19:49:29.662199    13 slave.cpp:1191] Asked to shut down framework
201405051517-67113388-5050-383-6805 by master@172.17.0.4:5050
> W0505 19:49:29.662256    13 slave.cpp:1206] Cannot shut down unknown
framework 201405051517-67113388-5050-383-6805
> I0505 19:49:30.662443    16 slave.cpp:1191] Asked to shut down framework
201405051517-67113388-5050-383-6806 by master@172.17.0.4:5050
> W0505 19:49:30.662489    16 slave.cpp:1206] Cannot shut down unknown
framework 201405051517-67113388-5050-383-6806
>
>
> Thanks in advance,
>
> Gerard.

Mime
View raw message