spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Ash <and...@andrewash.com>
Subject Re: Pre-build Spark for Windows 8.1
Date Sun, 08 Dec 2013 07:27:22 GMT
Thanks for the info Matei.  Seems like lots of other users of Akka have
similar issues -- maybe at some point in the future it'll be worth making
this a bit more flexible, but there are more important places to spend time
right now.


On Fri, Dec 6, 2013 at 12:06 PM, Matei Zaharia <matei.zaharia@gmail.com>wrote:

> Hey Andrew, unfortunately I don’t know how easy this is. Maybe future
> versions of Akka have it. We can certainly ask them to do it in general but
> I imagine there are some use cases where they wanted this behavior.
>
> Matei
>
>
> On Dec 5, 2013, at 2:49 PM, Andrew Ash <andrew@andrewash.com> wrote:
>
> Speaking of akka and host sensitivity...  How much have you hacked on akka
> to get it to support all of: myhost.mydomain.int, myhost, and 10.1.1.1?
>  It's kind of a pain to get the Spark URL to exactly match.  I'm wondering
> if there are usability gains that could be made here or if we're pretty
> stuck.
>
>
> On Thu, Dec 5, 2013 at 2:43 PM, Matei Zaharia <matei.zaharia@gmail.com>wrote:
>
>> Hi,
>>
>> When you launch the worker, try using spark://ADRIBONA-DEV-1:7077 as the
>> URL (uppercase instead of lowercase). Unfortunately Akka is very specific
>> about seeing hostnames written in the same way on each node, or else it
>> thinks the message is for another machine!
>>
>> Matei
>>
>> On Dec 5, 2013, at 8:27 AM, Adrian Bonar <Adrian.Bonar@microsoft.com>
>> wrote:
>>
>>  The master starts up now as expected but the workers are unable to
>> connect to the master. It looks like the master is refusing the connection
>> messages but I’m not sure why. The first two error lines below are from
>> trying to connect a worker from a separate machine and the last two error
>> lines are from trying to connect a worker on the same machine as the
>> master. I verified that the workers do not show up in the master’s web ui.
>>
>> MASTER:
>> D:\spark>spark-class org.apache.spark.deploy.master.Master
>> 13/12/05 08:08:34 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
>> 13/12/05 08:08:34 INFO master.Master: Starting Spark master at
>> spark://ADRIBONA-DEV-1:7077
>> 13/12/05 08:08:34 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/metrics/master/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/metrics/applications/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/static,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/app/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/app,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/json,null}
>> 13/12/05 08:08:34 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{*,null}
>> 13/12/05 08:08:34 INFO server.AbstractConnector: Started
>> SelectChannelConnector@0.0.0.0:8088
>> 13/12/05 08:08:34 INFO ui.MasterWebUI: Started Master web UI at
>> http://ADRIBONA-DEV-1:8088 <http://adribona-dev-1:8088/>
>> 13/12/05 08:09:15 *ERROR* NettyRemoteTransport(null): dropping message
>> RegisterWorker(worker-20131205080912-ADRIBONA-SFC-1-19280,ADRIBONA-SFC-1,19280,2,512,8081,ADRIBONA-SFC-1)
>> for non-local recipient
>> akka://sparkMaster@adribona-dev-1:7077/user/Master at
>> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
>> akka://sparkMaster@ADRIBONA-DEV-1:7077
>> 13/12/05 08:09:15 *ERROR* NettyRemoteTransport(null): dropping message
>> DaemonMsgWatch(Actor[
>> akka://sparkWorker@ADRIBONA-SFC-1:19280/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master])
>> for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at
>> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
>> akka://sparkMaster@ADRIBONA-DEV-1:7077
>> 13/12/05 08:18:46 *ERROR* NettyRemoteTransport(null): dropping message
>> RegisterWorker(worker-20131205081846-ADRIBONA-DEV-1-13521,ADRIBONA-DEV-1,13521,8,31670,8081,ADRIBONA-DEV-1)
>> for non-local recipient
>> akka://sparkMaster@adribona-dev-1:7077/user/Master at
>> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
>> akka://sparkMaster@ADRIBONA-DEV-1:7077
>> 13/12/05 08:18:46 *ERROR* NettyRemoteTransport(null): dropping message
>> DaemonMsgWatch(Actor[
>> akka://sparkWorker@ADRIBONA-DEV-1:13521/user/Worker],Actor[akka://sparkMaster@adribona-dev-1:7077/user/Master])
>> for non-local recipient akka://sparkMaster@adribona-dev-1:7077/remote at
>> akka://sparkMaster@ADRIBONA-DEV-1:7077 local is
>> akka://sparkMaster@ADRIBONA-DEV-1:7077
>>
>> WORKER:
>> D:\spark>spark-class.cmd org.apache.spark.deploy.worker.Worker
>> spark://adribona-dev-1:7077
>> 13/12/05 08:18:46 INFO slf4j.Slf4jEventHandler: Slf4jEventHandler started
>> 13/12/05 08:18:46 INFO worker.Worker: Starting Spark worker
>> ADRIBONA-DEV-1:13521 with 8 cores, 30.9 GB RAM
>> 13/12/05 08:18:46 INFO worker.Worker: Spark home: D:\spark
>> 13/12/05 08:18:46 INFO server.Server: jetty-7.x.y-SNAPSHOT
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/metrics/json,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/static,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/log,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/logPage,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{/json,null}
>> 13/12/05 08:18:46 INFO handler.ContextHandler: started
>> o.e.j.s.h.ContextHandler{*,null}
>> 13/12/05 08:18:46 INFO server.AbstractConnector: Started
>> SelectChannelConnector@0.0.0.0:8081
>> 13/12/05 08:18:46 INFO ui.WorkerWebUI: Started Worker web UI at
>> http://ADRIBONA-DEV-1:8081 <http://adribona-dev-1:8081/>
>> 13/12/05 08:18:46 INFO worker.Worker: Connecting to master
>> spark://adribona-dev-1:7077
>>
>>
>> --Adrian
>>
>> *From:* Adrian Bonar [mailto:Adrian.Bonar@microsoft.com<Adrian.Bonar@microsoft.com>
>> ]
>> *Sent:* Thursday, December 5, 2013 7:49 AM
>> *To:* user@spark.incubator.apache.org
>> *Subject:* RE: Pre-build Spark for Windows 8.1
>>
>>  Excellent! Thank you, Matei.
>>
>> *From:* Matei Zaharia [mailto:matei.zaharia@gmail.com<matei.zaharia@gmail.com>
>> ]
>> *Sent:* Wednesday, December 4, 2013 4:26 PM
>> *To:* user@spark.incubator.apache.org
>> *Subject:* Re: Pre-build Spark for Windows 8.1
>>
>>  Hey Adrian,
>>
>> Ideally you shouldn’t use Cygwin to run on Windows — use the .cmd scripts
>> we provide instead. Cygwin might be made to work but we haven’t tried to do
>> this so far so it’s not supported. If you can fix it, that would of course
>> be welcome.
>>
>> Also, the deploy scripts don’t work on Windows — we assumed that people
>> would mostly use Windows for local development and testing. However, if you
>> do want to launch a cluster, you can use spark-class.cmd manually to launch
>> a Master and then Workers. Use the commands in the first part of
>> http://spark.incubator.apache.org/docs/latest/spark-standalone.html.
>>
>> Matei
>>
>> On Dec 4, 2013, at 2:31 PM, Adrian Bonar <Adrian.Bonar@microsoft.com>
>> wrote:
>>
>>
>>
>> Separate from my previous thread about *building* a distribution of
>> Spark on Win8, I am also trying to run the pre-build binaries with little
>> success. I downloaded and extract “spark-0.8.0-incubating-bin-hadoop1” to
>> d:\spark and attempted to start a master with the following error:
>>
>> $ sh bin/start-master.sh
>> starting org.apache.spark.deploy.master.Master, logging to
>> /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
>> failed to launch org.apache.spark.deploy.master.Master:
>>
>>   Error: Could not find or load main class
>> org.apache.spark.deploy.master.Master
>> full log in
>> /cygdrive/d/spark/bin/../logs/spark-adribona-org.apache.spark.deploy.master.Master-1-ADRIBONA-DEV-1.out
>>
>> --Adrian (again. J)
>>
>>
>>
>
>

Mime
View raw message