spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Lee <alee...@hotmail.com>
Subject RE: spark-shell driver interacting with Workers in YARN mode - firewall blocking communication
Date Fri, 02 May 2014 20:15:07 GMT
Hi Yana, 
I did. I configured the the port in spark-env.sh, the problem is not the driver port which
is fixed.it's the Workers port that are dynamic every time when they are launched in the YARN
container. :-(
Any idea how to restrict the 'Workers' port range?

Date: Fri, 2 May 2014 14:49:23 -0400
Subject: Re: spark-shell driver interacting with Workers in YARN mode - firewall blocking
communication
From: yana.kadiyska@gmail.com
To: user@spark.apache.org

I think what you want to do is set spark.driver.port to a fixed port.


On Fri, May 2, 2014 at 1:52 PM, Andrew Lee <alee526@hotmail.com> wrote:




Hi All,
I encountered this problem when the firewall is enabled between the spark-shell and the Workers.
When I launch spark-shell in yarn-client mode, I notice that Workers on the YARN containers
are trying to talk to the driver (spark-shell), however, the firewall is not opened and caused
timeout.

For the Workers, it tries to open listening ports on 54xxx for each Worker? Is the port random
in such case?What will be the better way to predict the ports so I can configure the firewall
correctly between the driver (spark-shell) and the Workers? Is there a range of ports we can
specify in the firewall/iptables?

Any ideas? 		 	   		  

 		 	   		  
Mime
View raw message