spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shailesh Birari <sbirar...@gmail.com>
Subject Spark 1.2 – How to change Default (Random) port ….
Date Thu, 22 Jan 2015 02:14:23 GMT
Hello,

Recently, I have upgraded my setup to Spark 1.2 from Spark 1.1.

I have 4 node Ubuntu Spark Cluster.
With Spark 1.1, I used to write Spark Scala program in Eclipse on my Windows
development host and submit the job on Ubuntu Cluster, from Eclipse (Windows
machine).

As on my network not all ports between Spark cluster and development machine
are open, I set spark process ports to valid ports. 
On Spark 1.1 this works perfectly.

When I try to run the same program with same user defined ports on Spark 1.2
cluster it gives me connection time out for port *56117*.

I referred the Spark 1.2 configuration page
(http://spark.apache.org/docs/1.2.0/configuration.html) but there are no new
ports mentioned.

*Here is my code for reference:*
   
		val conf = new SparkConf()
        					.setMaster(sparkMaster)
        					.setAppName("Spark SVD")
        					.setSparkHome("/usr/local/spark")
        					.setJars(jars)
   					      .set("spark.driver.host", "consb2a")		//Windows host
(Development machine)
        					.set("spark.driver.port", "51810")
        					.set("spark.fileserver.port", "51811")
        					.set("spark.broadcast.port", "51812")
        					.set("spark.replClassServer.port", "51813")
        					.set("spark.blockManager.port", "51814")
        					.set("spark.executor.port", "51815")
        					.set("spark.executor.memory", "2g")
                  				.set("spark.driver.memory", "4g")
    val sc = new SparkContext(conf)

*Here is Exception:*
15/01/21 15:44:08 INFO BlockManagerMasterActor: Registering block manager
wynchcs217.wyn.cnw.co.nz:37173 with 1059.9 MB RAM, BlockManagerId(2,
wynchcs217.wyn.cnw.co.nz, 37173)
15/01/21 15:44:08 INFO BlockManagerMasterActor: Registering block manager
wynchcs219.wyn.cnw.co.nz:53850 with 1059.9 MB RAM, BlockManagerId(1,
wynchcs219.wyn.cnw.co.nz, 53850)
15/01/21 15:44:08 INFO BlockManagerMasterActor: Registering block manager
wynchcs220.wyn.cnw.co.nz:35670 with 1060.3 MB RAM, BlockManagerId(0,
wynchcs220.wyn.cnw.co.nz, 35670)
15/01/21 15:44:08 INFO BlockManagerMasterActor: Registering block manager
wynchcs218.wyn.cnw.co.nz:46890 with 1059.9 MB RAM, BlockManagerId(3,
wynchcs218.wyn.cnw.co.nz, 46890)
15/01/21 15:52:23 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
wynchcs217.wyn.cnw.co.nz): java.io.IOException: Connecting to
CONSB2A.cnw.co.nz/143.96.130.27:56117 timed out (120000 ms)
	at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:188)
	at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:156)
	at
org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:78)
	at
org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
	at
org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43)
	at
org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:701)

15/01/21 15:52:23 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID
2, wynchcs220.wyn.cnw.co.nz, NODE_LOCAL, 1366 bytes)
15/01/21 15:55:35 INFO TaskSchedulerImpl: Cancelling stage 0
15/01/21 15:55:35 INFO TaskSchedulerImpl: Stage 0 was cancelled
15/01/21 15:55:35 INFO DAGScheduler: Job 0 failed: count at
RowMatrix.scala:76, took 689.331309 s
Exception in thread "main" org.apache.spark.SparkException: Job 0 cancelled
because Stage 0 was cancelled


Can you please let me know how can I define the port 56117 to some other
port ?

Thanks,
  Shailesh





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-2-How-to-change-Default-Random-port-tp21306.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message