spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Clasen <scott.cla...@gmail.com>
Subject Spark Streaming + Kafka + Mesos/Marathon strangeness
Date Wed, 26 Mar 2014 19:18:19 GMT
I have a mesos cluster which runs marathon. 

I am using marathon to launch a long running spark streaming job which
consumes a Kafka Input Stream.

With one worker node in the cluster, I can successsfully launch the driver
job in marathon, which in turn launches a task in mesos via spark (spark is
using the coarse mode driver), which consumes just fine from kafka.

When I add nodes to the cluster, and start the driver job the first spark
mesos task runs fine for a few minutes, then exits (exit code 1) and the
tasks that are subsequently launched just sit there doing nothing, with this
being the last log output


14/03/26 19:06:47 INFO slf4j.Slf4jLogger: Slf4jLogger started
14/03/26 19:06:47 INFO Remoting: Starting remoting
14/03/26 19:06:47 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkExecutor@ip-xxx.ec2.internaxl:34488]
14/03/26 19:06:47 INFO Remoting: Remoting now listens on addresses:
[akka.tcp://sparkExecutor@xxxx.ec2.internal:34488]
14/03/26 19:06:47 INFO executor.CoarseGrainedExecutorBackend: Connecting to
driver: akka.tcp://spark@xxxx.ec2.internal:35332/user/CoarseGrainedScheduler
14/03/26 19:06:47 INFO executor.CoarseGrainedExecutorBackend: Successfully
registered with driver
14/03/26 19:06:48 INFO slf4j.Slf4jLogger: Slf4jLogger started
14/03/26 19:06:48 INFO Remoting: Starting remoting
14/03/26 19:06:48 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://spark@xxxx.ec2.internal:59070]
14/03/26 19:06:48 INFO Remoting: Remoting now listens on addresses:
[akka.tcp://spark@xxxx.ec2.internal:59070]
14/03/26 19:06:48 INFO spark.SparkEnv: Connecting to BlockManagerMaster:
akka.tcp://spark@xxxx.ec2.internal:35332/user/BlockManagerMaster
14/03/26 19:06:48 INFO storage.DiskBlockManager: Created local directory at
/tmp/spark-local-20140326190648-fa77
14/03/26 19:06:48 INFO storage.MemoryStore: MemoryStore started with
capacity 294.4 MB.
14/03/26 19:06:48 INFO network.ConnectionManager: Bound socket to port 55018
with id = ConnectionManagerId(xxxx.ec2.internal,55018)
14/03/26 19:06:48 INFO storage.BlockManagerMaster: Trying to register
BlockManager
14/03/26 19:06:48 INFO storage.BlockManagerMaster: Registered BlockManager
14/03/26 19:06:48 INFO spark.SparkEnv: Connecting to MapOutputTracker:
akka.tcp://spark@xxxx.ec2.internal:35332/user/MapOutputTracker
14/03/26 19:06:48 INFO spark.HttpFileServer: HTTP File server directory is
/tmp/spark-15f43b7b-6f7c-48dd-8bd8-5663a00fd314
14/03/26 19:06:48 INFO spark.HttpServer: Starting HTTP Server
14/03/26 19:06:48 INFO server.Server: jetty-7.x.y-SNAPSHOT
14/03/26 19:06:48 INFO server.AbstractConnector: Started
SocketConnector@0.0.0.0:45286



Any pointers as to un-stick this??






--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-Mesos-Marathon-strangeness-tp3285.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message