spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: IPv6 support
Date Wed, 24 Jun 2015 18:07:21 GMT
+Dev list

Thanks
Best Regards

On Wed, Jun 24, 2015 at 11:26 PM, Kevin Liu <kevinliu@fb.com> wrote:

>  Continuing this thread beyond standalone - onto clusters, does anyone
> have experience successfully running any Spark cluster on IPv6 *only*
> (not dual stack) machines? More companies are moving to IPv6 and some such
> as Facebook are only allocating new clusters on IPv6 only network, so this
> is getting more relevant.
>
>  YARN still doesn’t support IPv6 per
> http://wiki.apache.org/hadoop/HadoopIPv6
>
>  Mesos is questionable per
> https://issues.apache.org/jira/browse/MESOS-1027 , did anyone get it
> working?
>
>  Standalone: Even though below worked in a single node mode, when I tried
> to connect to a remote master - it failed with the following, nor did it
> work with IPv6 address directly like "./bin/spark-shell --master
> spark://[2401:db00:2030:709b:face:0:9:0]:7078"
> client side:
>
> [root@dispark002.ash3 ~/spark-1.4.0-bin-hadoop2.6]# ./bin/spark-shell
> --master spark://dispark001:7078
>
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
>
> log4j:WARN Please initialize the log4j system properly.
>
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
>
> 15/06/24 10:34:03 INFO SecurityManager: Changing view acls to: root
>
> 15/06/24 10:34:03 INFO SecurityManager: Changing modify acls to: root
>
> 15/06/24 10:34:03 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(root); users
> with modify permissions: Set(root)
>
> 15/06/24 10:34:03 INFO HttpServer: Starting HTTP Server
>
> 15/06/24 10:34:03 INFO Utils: Successfully started service 'HTTP class
> server' on port 49189.
>
> Welcome to
>
>       ____              __
>
>      / __/__  ___ _____/ /__
>
>     _\ \/ _ \/ _ `/ __/  '_/
>
>    /___/ .__/\_,_/_/ /_/\_\   version 1.4.0
>
>       /_/
>
>
>  Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
> 1.8.0_25)
>
> Type in expressions to have them evaluated.
>
> Type :help for more information.
>
> 15/06/24 10:34:05 INFO SparkContext: Running Spark version 1.4.0
>
> 15/06/24 10:34:05 INFO SecurityManager: Changing view acls to: root
>
> 15/06/24 10:34:05 INFO SecurityManager: Changing modify acls to: root
>
> 15/06/24 10:34:05 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(root); users
> with modify permissions: Set(root)
>
> 15/06/24 10:34:06 INFO Slf4jLogger: Slf4jLogger started
>
> 15/06/24 10:34:06 INFO Remoting: Starting remoting
>
> 15/06/24 10:34:06 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@dispark002:59150]
>
> 15/06/24 10:34:06 INFO Utils: Successfully started service 'sparkDriver'
> on port 59150.
>
> 15/06/24 10:34:06 INFO SparkEnv: Registering MapOutputTracker
>
> 15/06/24 10:34:06 INFO SparkEnv: Registering BlockManagerMaster
>
> 15/06/24 10:34:06 INFO DiskBlockManager: Created local directory at
> /tmp/spark-b4248e03-80c2-4d54-b3af-5044c8228f68/blockmgr-bb240921-31bf-48da-b96a-7120f118d002
>
> 15/06/24 10:34:06 INFO MemoryStore: MemoryStore started with capacity
> 265.1 MB
>
> 15/06/24 10:34:06 INFO HttpFileServer: HTTP File server directory is
> /tmp/spark-b4248e03-80c2-4d54-b3af-5044c8228f68/httpd-a7cbeb43-aefd-4da8-8df2-89a528b35c9e
>
> 15/06/24 10:34:06 INFO HttpServer: Starting HTTP Server
>
> 15/06/24 10:34:06 INFO Utils: Successfully started service 'HTTP file
> server' on port 57293.
>
> 15/06/24 10:34:06 INFO SparkEnv: Registering OutputCommitCoordinator
>
> 15/06/24 10:34:06 INFO Utils: Successfully started service 'SparkUI' on
> port 4040.
>
> 15/06/24 10:34:06 INFO SparkUI: Started SparkUI at http://
> [2401:db00:2030:709b:face:0:f:0]:4040
>
> 15/06/24 10:34:06 INFO AppClient$ClientActor: Connecting to master
> akka.tcp://sparkMaster@dispark001:7078/user/Master...
>
> 15/06/24 10:34:06 INFO SparkDeploySchedulerBackend: Connected to Spark
> cluster with app ID app-20150624103406-0004
>
> 15/06/24 10:34:06 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 64775.
>
> 15/06/24 10:34:06 INFO NettyBlockTransferService: Server created on 64775
>
> 15/06/24 10:34:06 ERROR SparkContext: Error initializing SparkContext.
>
> java.lang.AssertionError: assertion failed: Expected hostname
>
> at scala.Predef$.assert(Predef.scala:179)
>
> at org.apache.spark.util.Utils$.checkHost(Utils.scala:882)
>
> at org.apache.spark.storage.BlockManagerId.<init>(BlockManagerId.scala:48)
>
> at org.apache.spark.storage.BlockManagerId$.apply(BlockManagerId.scala:107)
>
> at org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:188)
>
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:502)
>
> at
> org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017)
>
> at $line3.$read$$iwC$$iwC.<init>(<console>:9)
>
> at $line3.$read$$iwC.<init>(<console>:18)
>
> at $line3.$read.<init>(<console>:20)
>
> at $line3.$read$.<init>(<console>:24)
>
> at $line3.$read$.<clinit>(<console>)
>
> at $line3.$eval$.<init>(<console>:7)
>
> at $line3.$eval$.<clinit>(<console>)
>
> at $line3.$eval.$print(<console>)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:483)
>
> at
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
>
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
>
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
>
> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
>
> at
> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
>
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
>
> at
> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:123)
>
> at
> org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:122)
>
> at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
>
> at
> org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:122)
>
> at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
>
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
>
> at
> org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:157)
>
> at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
>
> at
> org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:106)
>
> at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
>
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991)
>
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
>
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
>
> at
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
>
> at org.apache.spark.repl.SparkILoop.org
> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
>
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
>
> at org.apache.spark.repl.Main$.main(Main.scala:31)
>
> at org.apache.spark.repl.Main.main(Main.scala)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:483)
>
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
>
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
>
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
>
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
>
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
>  server side:
>
> 15/06/24 10:34:06 INFO Master: Registering app Spark shell
>
> 15/06/24 10:34:06 INFO Master: Registered app Spark shell with ID
> app-20150624103406-0004
>
> 15/06/24 10:34:06 INFO Master: Received unregister request from
> application app-20150624103406-0004
>
> 15/06/24 10:34:06 INFO Master: Removing app app-20150624103406-0004
>
> 15/06/24 10:34:07 INFO Master: akka.tcp://sparkDriver@dispark002:59150
> got disassociated, removing it.
>
>
>
>   From: Kevin Liu <kevinliu@fb.com>
> Date: Wednesday, June 17, 2015 at 11:21 AM
> To: Akhil Das <akhil@sigmoidanalytics.com>
>
> Cc: "user@spark.apache.org" <user@spark.apache.org>
> Subject: Re: IPv6 support
>
>   You Sir - are a genius. Thank you so much, works now…
>
>  Still wondering why I have to do this for IPv6 only machines when the
> default just works on dual-stack machines, but I am happy enough for now.
>
>  Kevin
>
>   From: Akhil Das <akhil@sigmoidanalytics.com>
> Date: Wednesday, June 17, 2015 at 12:33 AM
> To: Kevin Liu <kevinliu@fb.com>
> Cc: "user@spark.apache.org" <user@spark.apache.org>
> Subject: Re: IPv6 support
>
>    If you look at this,
>
>   15/06/16 22:25:14 INFO Executor: Starting executor ID driver on host
> localhost
>
> 15/06/16 22:25:14 ERROR SparkContext: Error initializing SparkContext.
>
> java.lang.AssertionError: assertion failed: Expected hostname
>
>  your spark.driver.host
> <https://github.com/apache/spark/blob/3c0156899dc1ec1f7dfe6d7c8af47fa6dc7d00bf/core/src/main/scala/org/apache/spark/util/RpcUtils.scala#L33>
is
> being set to localhost which fails host.indexOf(':') == -1
> <https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/util/Utils.scala#L882>.
> Try setting your spark.driver.host to dispark001.ash3 (from whichever
> machine you are running the code)
>
>
>
>
>  Thanks
> Best Regards
>
> On Wed, Jun 17, 2015 at 10:57 AM, Kevin Liu <kevinliu@fb.com> wrote:
>
>>  Thanks Akhil, it seems that what you said should have fixed it.
>>
>>  1) There was a known issue directly related to below, but it has been
>> fixed in 1.4 https://issues.apache.org/jira/browse/SPARK-6440
>> <https://urldefense.proofpoint.com/v1/url?u=https://issues.apache.org/jira/browse/SPARK-6440&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=YviX1%2F1vaAZK%2BrqaSzu%2FMg%3D%3D%0A&m=06CDmFdO8vUzzjbqnMBz2CS55qwEN6lPy%2FypbKwNdow%3D%0A&s=464bc64957c6ddd8bb506c8ec449f3c26f8c6f4482dc6ea7356df4e0cb5c98e9>
>> 2) Now, with 1.4 - I see the following errors - even after I setting
>> SPARK_MASTER_IP - your description seems to be right on, any more thoughts?
>>
>>   [root@dispark001.ash3 ~/spark-1.4.0-bin-hadoop2.6]# ping6
>> dispark001.ash3
>>
>> PING dispark001.ash3(dispark001.ash3.facebook.com) 56 data bytes
>>
>> 64 bytes from dispark001.ash3.facebook.com: icmp_seq=1 ttl=64 time=0.012
>> ms
>>
>> 64 bytes from dispark001.ash3.facebook.com: icmp_seq=2 ttl=64 time=0.021
>> ms
>>
>> 64 bytes from dispark001.ash3.facebook.com: icmp_seq=3 ttl=64 time=0.011
>> ms
>>
>> ^C
>>
>> --- dispark001.ash3 ping statistics ---
>>
>> 3 packets transmitted, 3 received, 0% packet loss, time 2113ms
>>
>> rtt min/avg/max/mdev = 0.011/0.014/0.021/0.006 ms
>>
>> [root@dispark001.ash3 ~/spark-1.4.0-bin-hadoop2.6]# export
>> SPARK_MASTER_IP="dispark001.ash3"
>>
>> [root@dispark001.ash3 ~/spark-1.4.0-bin-hadoop2.6]# ./bin/run-example
>> SparkPi 10
>>
>> Using Spark's default log4j profile:
>> org/apache/spark/log4j-defaults.properties
>>
>> 15/06/16 22:25:13 INFO SparkContext: Running Spark version 1.4.0
>>
>> 15/06/16 22:25:13 WARN NativeCodeLoader: Unable to load native-hadoop
>> library for your platform... using builtin-java classes where applicable
>>
>> 15/06/16 22:25:13 INFO SecurityManager: Changing view acls to: root
>>
>> 15/06/16 22:25:13 INFO SecurityManager: Changing modify acls to: root
>>
>> 15/06/16 22:25:13 INFO SecurityManager: SecurityManager: authentication
>> disabled; ui acls disabled; users with view permissions: Set(root); users
>> with modify permissions: Set(root)
>>
>> 15/06/16 22:25:13 INFO Slf4jLogger: Slf4jLogger started
>>
>> 15/06/16 22:25:13 INFO Remoting: Starting remoting
>>
>> 15/06/16 22:25:13 INFO Remoting: Remoting started; listening on addresses
>> :[akka.tcp://sparkDriver@2401:db00:2030:709b:face:0:9:0:50916]
>>
>> 15/06/16 22:25:13 INFO Utils: Successfully started service 'sparkDriver'
>> on port 50916.
>>
>> 15/06/16 22:25:13 INFO SparkEnv: Registering MapOutputTracker
>>
>> 15/06/16 22:25:13 INFO SparkEnv: Registering BlockManagerMaster
>>
>> 15/06/16 22:25:13 INFO DiskBlockManager: Created local directory at
>> /tmp/spark-325ae594-5b00-42e2-bd47-70c92c4e5f66/blockmgr-4bcbf896-03ad-4b11-8db5-a6eaa5f0222b
>>
>> 15/06/16 22:25:13 INFO MemoryStore: MemoryStore started with capacity
>> 265.1 MB
>>
>> 15/06/16 22:25:13 INFO HttpFileServer: HTTP File server directory is
>> /tmp/spark-325ae594-5b00-42e2-bd47-70c92c4e5f66/httpd-2b6a5162-0686-4cc5-accb-1bb66fddf705
>>
>> 15/06/16 22:25:13 INFO HttpServer: Starting HTTP Server
>>
>> 15/06/16 22:25:13 INFO Utils: Successfully started service 'HTTP file
>> server' on port 35895.
>>
>> 15/06/16 22:25:13 INFO SparkEnv: Registering OutputCommitCoordinator
>>
>> 15/06/16 22:25:13 INFO Utils: Successfully started service 'SparkUI' on
>> port 4040.
>>
>> 15/06/16 22:25:13 INFO SparkUI: Started SparkUI at
>> http://[2401:db00:2030:709b:face:0:9:0]:4040
>>
>> 15/06/16 22:25:14 INFO SparkContext: Added JAR
>> file:/root/spark-1.4.0-bin-hadoop2.6/lib/spark-examples-1.4.0-hadoop2.6.0.jar
>> at
>> http://[2401:db00:2030:709b:face:0:9:0]:35895/jars/spark-examples-1.4.0-hadoop2.6.0.jar
>> with timestamp 1434518714122
>>
>> 15/06/16 22:25:14 INFO Executor: Starting executor ID driver on host
>> localhost
>>
>> 15/06/16 22:25:14 ERROR SparkContext: Error initializing SparkContext.
>>
>> java.lang.AssertionError: assertion failed: Expected hostname
>>
>> at scala.Predef$.assert(Predef.scala:179)
>>
>> at org.apache.spark.util.Utils$.checkHost(Utils.scala:882)
>>
>> at org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:35)
>>
>> at org.apache.spark.executor.Executor.<init>(Executor.scala:413)
>>
>> at
>> org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalBackend.scala:53)
>>
>> at
>> org.apache.spark.scheduler.local.LocalBackend.start(LocalBackend.scala:103)
>>
>> at
>> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
>>
>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:497)
>>
>> at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:28)
>>
>> at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:483)
>>
>> at
>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
>>
>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
>>
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
>>
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
>>
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> 15/06/16 22:25:14 INFO SparkUI: Stopped Spark web UI at
>> http://[2401:db00:2030:709b:face:0:9:0]:4040
>>
>> 15/06/16 22:25:14 INFO DAGScheduler: Stopping DAGScheduler
>>
>> 15/06/16 22:25:14 ERROR SparkContext: Error stopping SparkContext after
>> init error.
>>
>> java.lang.NullPointerException
>>
>> at
>> org.apache.spark.scheduler.local.LocalBackend.stop(LocalBackend.scala:107)
>>
>> at
>> org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:416)
>>
>> at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1404)
>>
>> at org.apache.spark.SparkContext.stop(SparkContext.scala:1642)
>>
>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:565)
>>
>> at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:28)
>>
>> at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:483)
>>
>> at
>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
>>
>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
>>
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
>>
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
>>
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> Exception in thread "main" java.lang.AssertionError: assertion failed:
>> Expected hostname
>>
>> at scala.Predef$.assert(Predef.scala:179)
>>
>> at org.apache.spark.util.Utils$.checkHost(Utils.scala:882)
>>
>> at org.apache.spark.util.RpcUtils$.makeDriverRef(RpcUtils.scala:35)
>>
>> at org.apache.spark.executor.Executor.<init>(Executor.scala:413)
>>
>> at
>> org.apache.spark.scheduler.local.LocalEndpoint.<init>(LocalBackend.scala:53)
>>
>> at
>> org.apache.spark.scheduler.local.LocalBackend.start(LocalBackend.scala:103)
>>
>> at
>> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
>>
>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:497)
>>
>> at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:28)
>>
>> at org.apache.spark.examples.SparkPi.main(SparkPi.scala)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:483)
>>
>> at
>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
>>
>> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
>>
>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
>>
>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
>>
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
>> 15/06/16 22:25:14 INFO DiskBlockManager: Shutdown hook called
>>
>> 15/06/16 22:25:14 INFO Utils: path =
>> /tmp/spark-325ae594-5b00-42e2-bd47-70c92c4e5f66/blockmgr-4bcbf896-03ad-4b11-8db5-a6eaa5f0222b,
>> already present as root for deletion.
>>
>> 15/06/16 22:25:14 INFO Utils: Shutdown hook called
>>
>> 15/06/16 22:25:14 INFO Utils: Deleting directory
>> /tmp/spark-325ae594-5b00-42e2-bd47-70c92c4e5f66
>>
>> [root@dispark001.ash3 ~/spark-1.4.0-bin-hadoop2.6]#
>>
>>
>>   From: Akhil Das <akhil@sigmoidanalytics.com>
>> Date: Monday, May 25, 2015 at 9:33 AM
>> To: Kevin Liu <kevinliu@fb.com>
>> Cc: "user@spark.apache.org" <user@spark.apache.org>
>> Subject: Re: IPv6 support
>>
>>    Hi Kevin,
>>
>>  Did you try adding a host name for the ipv6? I have a few ipv6 boxes,
>> spark failed for me when i use just the ipv6 addresses, but it works fine
>> when i use the host names.
>>
>>  Here's an entry in my /etc/hosts:
>>
>>  2607:5300:0100:0200:0000:0000:0000:0a4d hacked.work
>>
>>
>>  My spark-env.sh file:
>>
>>  export SPARK_MASTER_IP="hacked.work"
>>
>>
>>  Here's the master listening on my v6:
>>
>>   [image: Inline image 1]
>>
>>
>>  The Master UI with running spark-shell:
>>
>>   [image: Inline image 2]
>>
>>
>>  I even ran a simple sc.parallelize(1 to 100).collect().
>>
>>
>>
>>  Thanks
>> Best Regards
>>
>> On Wed, May 20, 2015 at 11:09 PM, Kevin Liu <kevinliu@fb.com> wrote:
>>
>>> Hello, I have to work with IPv6 only servers and when I installed the
>>> 1.3.1 hadoop 2.6 build, I couldn¹t get the example to run due to IPv6
>>> issues (errors below). I tried to add the
>>> -Djava.net.preferIPv6Addresses=true setting but it still doesn¹t work. A
>>> search on Spark¹s support for IPv6 is inconclusive. Can someone help
>>> clarify the current status for IPv6?
>>>
>>> Thanks
>>> Kevin
>>>
>>>
>>> ‹‹ errors ‹
>>>
>>> 5/05/20 10:17:30 INFO Executor: Fetching
>>>
>>> http://2401:db00:2030:709b:face:0:9:0:51453/jars/spark-examples-1.3.1-hadoo
>>> <https://urldefense.proofpoint.com/v1/url?u=http://2401:db00:2030:709b:face:0:9:0:51453/jars/spark-examples-1.3.1-hadoo&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=YviX1%2F1vaAZK%2BrqaSzu%2FMg%3D%3D%0A&m=06CDmFdO8vUzzjbqnMBz2CS55qwEN6lPy%2FypbKwNdow%3D%0A&s=36fe35441ecafdbc99ac8e5605e91d0a4ea88855d27d1a964e55e04ae65c7fde>
>>> p2.6.0.jar with timestamp 1432142250197
>>> 15/05/20 10:17:30 INFO Executor: Fetching
>>>
>>> http://2401:db00:2030:709b:face:0:9:0:51453/jars/spark-examples-1.3.1-hadoo
>>> <https://urldefense.proofpoint.com/v1/url?u=http://2401:db00:2030:709b:face:0:9:0:51453/jars/spark-examples-1.3.1-hadoo&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=YviX1%2F1vaAZK%2BrqaSzu%2FMg%3D%3D%0A&m=06CDmFdO8vUzzjbqnMBz2CS55qwEN6lPy%2FypbKwNdow%3D%0A&s=36fe35441ecafdbc99ac8e5605e91d0a4ea88855d27d1a964e55e04ae65c7fde>
>>> p2.6.0.jar with timestamp 1432142250197
>>> 15/05/20 10:17:30 ERROR Executor: Exception in task 5.0 in stage 0.0 (TID
>>> 5)
>>> java.net.MalformedURLException: For input string:
>>> "db00:2030:709b:face:0:9:0:51453"
>>>         at java.net.URL.<init>(URL.java:620)
>>>         at java.net.URL.<init>(URL.java:483)
>>>         at java.net.URL.<init>(URL.java:432)
>>>         at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:603)
>>>         at org.apache.spark.util.Utils$.fetchFile(Utils.scala:431)
>>>         at
>>>
>>> org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Execu
>>> tor$$updateDependencies$5.apply(Executor.scala:374)
>>>         at
>>>
>>> org.apache.spark.executor.Executor$$anonfun$org$apache$spark$executor$Execu
>>> tor$$updateDependencies$5.apply(Executor.scala:366)
>>>         at
>>>
>>> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(Traver
>>> sableLike.scala:772)
>>>         at
>>>
>>> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
>>>         at
>>>
>>> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
>>>         at
>>>
>>> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
>>>         at
>>> scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
>>>         at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
>>>         at
>>>
>>> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:7
>>> 71)
>>>         at
>>> org.apache.spark.executor.Executor.org
>>> <https://urldefense.proofpoint.com/v1/url?u=http://org.apache.spark.executor.Executor.org&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=YviX1%2F1vaAZK%2BrqaSzu%2FMg%3D%3D%0A&m=cToWyWSkkQGPchtZEHgZAymvCC%2FYOX8btPSeh%2Bth5wM%3D%0A&s=69d2377deecbf9077810fa58426e84fc72e54932d7bf06064e130e9a3ac6af04>
>>> $apache$spark$executor$Executor$$upda
>>> teDependencies(Executor.scala:366)
>>>         at
>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:184)
>>>         at
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1
>>> 142)
>>>         at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
>>> 617)
>>>         at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.lang.NumberFormatException: For input string:
>>> "db00:2030:709b:face:0:9:0:51453"
>>>         at
>>>
>>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:6
>>> 5)
>>>         at java.lang.Integer.parseInt(Integer.java:580)
>>>         at java.lang.Integer.parseInt(Integer.java:615)
>>>         at java.net.URLStreamHandler.parseURL(URLStreamHandler.java:216)
>>>         at java.net.URL.<init>(URL.java:615)
>>>         ... 18 more
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>

Mime
View raw message