spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yana Kadiyska <yana.kadiy...@gmail.com>
Subject Re: spark-shell driver interacting with Workers in YARN mode - firewall blocking communication
Date Fri, 02 May 2014 18:49:23 GMT
I think what you want to do is set spark.driver.port to a fixed port.


On Fri, May 2, 2014 at 1:52 PM, Andrew Lee <alee526@hotmail.com> wrote:

> Hi All,
>
> I encountered this problem when the firewall is enabled between the
> spark-shell and the Workers.
>
> When I launch spark-shell in yarn-client mode, I notice that Workers on
> the YARN containers are trying to talk to the driver (spark-shell),
> however, the firewall is not opened and caused timeout.
>
> For the Workers, it tries to open listening ports on 54xxx for each
> Worker? Is the port random in such case?
> What will be the better way to predict the ports so I can configure the
> firewall correctly between the driver (spark-shell) and the Workers? Is
> there a range of ports we can specify in the firewall/iptables?
>
> Any ideas?
>

Mime
View raw message