spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Davidson <ilike...@gmail.com>
Subject Re: spark with docker: errors with akka, NAT?
Date Tue, 17 Jun 2014 23:58:17 GMT
I remember having to do a similar thing in the spark docker scripts for
testing purposes. Were you able to modify the /etc/hosts directly? I
remember issues with that as docker apparently mounts it as part of its
read-only filesystem.


On Tue, Jun 17, 2014 at 4:36 PM, Mohit Jaggi <mohitjaggi@gmail.com> wrote:

> It was a DNS issue. AKKA apparently uses the hostname of the endpoints and
> hence they need to be resolvable. In my case the hostname of the docker
> container was a randomly generated string and was not resolvable. I added a
> workaround (entry in etc/hosts file of spark master) for now. If anyone can
> point to a more elegant solution, that would be awesome!
>
>
> On Tue, Jun 17, 2014 at 3:48 PM, Mohit Jaggi <mohitjaggi@gmail.com> wrote:
>
>> I am using cutting edge code from git but doing my own sbt assembly.
>>
>>
>> On Mon, Jun 16, 2014 at 10:28 PM, Andre Schumacher <
>> schumach@icsi.berkeley.edu> wrote:
>>
>>>
>>> Hi,
>>>
>>> are you using the amplab/spark-1.0.0 images from the global registry?
>>>
>>> Andre
>>>
>>> On 06/17/2014 01:36 AM, Mohit Jaggi wrote:
>>> > Hi Folks,
>>> >
>>> > I am having trouble getting spark driver running in docker. If I run a
>>> > pyspark example on my mac it works but the same example on a docker
>>> image
>>> > (Via boot2docker) fails with following logs. I am pointing the spark
>>> driver
>>> > (which is running the example) to a spark cluster (driver is not part
>>> of
>>> > the cluster). I guess this has something to do with docker's networking
>>> > stack (it may be getting NAT'd) but I am not sure why (if at all) the
>>> > spark-worker or spark-master is trying to create a new TCP connection
>>> to
>>> > the driver, instead of responding on the connection initiated by the
>>> driver.
>>> >
>>> > I would appreciate any help in figuring this out.
>>> >
>>> > Thanks,
>>> >
>>> > Mohit.
>>> >
>>> > --------logs--------
>>> >
>>> > Spark Executor Command: "java" "-cp"
>>> >
>>> "::/home/ayasdi/spark/conf:/home/xxxx/spark/assembly/target/scala-2.10/spark-assembly-1.0.0-hadoop1.0.4.jar"
>>> > "-Xms2g" "-Xmx2g" "-Xms512M" "-Xmx512M"
>>> > "org.apache.spark.executor.CoarseGrainedExecutorBackend"
>>> > "akka.tcp://spark@fc31887475e3:43921/user/CoarseGrainedScheduler" "1"
>>> > "cobalt" "24" "akka.tcp://sparkWorker@aaaa:33952/user/Worker"
>>> > "app-20140616152201-0021"
>>> >
>>> > ========================================
>>> >
>>> >
>>> > log4j:WARN No appenders could be found for logger
>>> > (org.apache.hadoop.conf.Configuration).
>>> >
>>> > log4j:WARN Please initialize the log4j system properly.
>>> >
>>> > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>> for
>>> > more info.
>>> >
>>> > 14/06/16 15:22:05 INFO SparkHadoopUtil: Using Spark's default log4j
>>> > profile: org/apache/spark/log4j-defaults.properties
>>> >
>>> > 14/06/16 15:22:05 INFO SecurityManager: Changing view acls to:
>>> ayasdi,root
>>> >
>>> > 14/06/16 15:22:05 INFO SecurityManager: SecurityManager: authentication
>>> > disabled; ui acls disabled; users with view permissions: Set(xxx, xxx)
>>> >
>>> > 14/06/16 15:22:05 INFO Slf4jLogger: Slf4jLogger started
>>> >
>>> > 14/06/16 15:22:05 INFO Remoting: Starting remoting
>>> >
>>> > 14/06/16 15:22:06 INFO Remoting: Remoting started; listening on
>>> addresses
>>> > :[akka.tcp://sparkExecutor@aaaa:33536]
>>> >
>>> > 14/06/16 15:22:06 INFO Remoting: Remoting now listens on addresses:
>>> > [akka.tcp://sparkExecutor@aaaa:33536]
>>> >
>>> > 14/06/16 15:22:06 INFO CoarseGrainedExecutorBackend: Connecting to
>>> driver:
>>> > akka.tcp://spark@fc31887475e3:43921/user/CoarseGrainedScheduler
>>> >
>>> > 14/06/16 15:22:06 INFO WorkerWatcher: Connecting to worker
>>> > akka.tcp://sparkWorker@aaaa:33952/user/Worker
>>> >
>>> > 14/06/16 15:22:06 WARN Remoting: Tried to associate with unreachable
>>> remote
>>> > address [akka.tcp://spark@fc31887475e3:43921]. Address is now gated
>>> for
>>> > 60000 ms, all messages to this address will be delivered to dead
>>> letters.
>>> >
>>> > 14/06/16 15:22:06 ERROR CoarseGrainedExecutorBackend: Driver
>>> Disassociated
>>> > [akka.tcp://sparkExecutor@aaaa:33536] ->
>>> [akka.tcp://spark@fc31887475e3:43921]
>>> > disassociated! Shutting down.
>>> >
>>>
>>>
>>
>

Mime
View raw message