spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andre Schumacher <schum...@icsi.berkeley.edu>
Subject Re: spark with docker: errors with akka, NAT?
Date Tue, 17 Jun 2014 05:28:15 GMT

Hi,

are you using the amplab/spark-1.0.0 images from the global registry?

Andre

On 06/17/2014 01:36 AM, Mohit Jaggi wrote:
> Hi Folks,
> 
> I am having trouble getting spark driver running in docker. If I run a
> pyspark example on my mac it works but the same example on a docker image
> (Via boot2docker) fails with following logs. I am pointing the spark driver
> (which is running the example) to a spark cluster (driver is not part of
> the cluster). I guess this has something to do with docker's networking
> stack (it may be getting NAT'd) but I am not sure why (if at all) the
> spark-worker or spark-master is trying to create a new TCP connection to
> the driver, instead of responding on the connection initiated by the driver.
> 
> I would appreciate any help in figuring this out.
> 
> Thanks,
> 
> Mohit.
> 
> --------logs--------
> 
> Spark Executor Command: "java" "-cp"
> "::/home/ayasdi/spark/conf:/home/xxxx/spark/assembly/target/scala-2.10/spark-assembly-1.0.0-hadoop1.0.4.jar"
> "-Xms2g" "-Xmx2g" "-Xms512M" "-Xmx512M"
> "org.apache.spark.executor.CoarseGrainedExecutorBackend"
> "akka.tcp://spark@fc31887475e3:43921/user/CoarseGrainedScheduler" "1"
> "cobalt" "24" "akka.tcp://sparkWorker@aaaa:33952/user/Worker"
> "app-20140616152201-0021"
> 
> ========================================
> 
> 
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.conf.Configuration).
> 
> log4j:WARN Please initialize the log4j system properly.
> 
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
> 
> 14/06/16 15:22:05 INFO SparkHadoopUtil: Using Spark's default log4j
> profile: org/apache/spark/log4j-defaults.properties
> 
> 14/06/16 15:22:05 INFO SecurityManager: Changing view acls to: ayasdi,root
> 
> 14/06/16 15:22:05 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(xxx, xxx)
> 
> 14/06/16 15:22:05 INFO Slf4jLogger: Slf4jLogger started
> 
> 14/06/16 15:22:05 INFO Remoting: Starting remoting
> 
> 14/06/16 15:22:06 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkExecutor@aaaa:33536]
> 
> 14/06/16 15:22:06 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://sparkExecutor@aaaa:33536]
> 
> 14/06/16 15:22:06 INFO CoarseGrainedExecutorBackend: Connecting to driver:
> akka.tcp://spark@fc31887475e3:43921/user/CoarseGrainedScheduler
> 
> 14/06/16 15:22:06 INFO WorkerWatcher: Connecting to worker
> akka.tcp://sparkWorker@aaaa:33952/user/Worker
> 
> 14/06/16 15:22:06 WARN Remoting: Tried to associate with unreachable remote
> address [akka.tcp://spark@fc31887475e3:43921]. Address is now gated for
> 60000 ms, all messages to this address will be delivered to dead letters.
> 
> 14/06/16 15:22:06 ERROR CoarseGrainedExecutorBackend: Driver Disassociated
> [akka.tcp://sparkExecutor@aaaa:33536] -> [akka.tcp://spark@fc31887475e3:43921]
> disassociated! Shutting down.
> 


Mime
View raw message