spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Satoshi Yamada <satoshi.yamada....@gmail.com>
Subject Port to open for submitting Spark on Yarn application
Date Mon, 04 Sep 2017 00:50:23 GMT
Hi,

In case we run Spark on Yarn in client mode, we have firewall for
Hadoop cluster,
and the client node is outside firewall, I think I have to open some ports
that Application Master uses.


I think the ports is specified by "spark.yarn.am.port" as document says.
https://spark.apache.org/docs/latest/running-on-yarn.html

But, according to the source code, spark.yarn.am.port is deprecated since 2.0.
https://github.com/apache/spark/commit/829cd7b8b70e65a91aa66e6d626bd45f18e0ad97

Does this mean we do not need to open particular ports of firewall for

Spark on Yarn?


Thanks,

Mime
View raw message