spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Satoshi Yamada <satoshi.yamada....@gmail.com>
Subject Re: Port to open for submitting Spark on Yarn application
Date Mon, 04 Sep 2017 05:13:07 GMT
Jerry,

Thanks for your comment.

On Mon, Sep 4, 2017 at 10:43 AM, Saisai Shao <sai.sai.shao@gmail.com> wrote:

> I think spark.yarn.am.port is not used any more, so you don't need to
> consider this.
>
> If you're running Spark on YARN, I think some YARN RM port to submit
> applications should also be reachable via firewall, as well as HDFS port to
> upload resources.
>
> Also in the Spark side, executors will be connected to driver via
> spark.driver.port, maybe you should also set a fixed port number for this
> and add to white list of firewall.
>
> Thanks
> Jerry
>
>
> On Mon, Sep 4, 2017 at 8:50 AM, Satoshi Yamada <
> satoshi.yamada.dev@gmail.com> wrote:
>
>> Hi,
>>
>> In case we run Spark on Yarn in client mode, we have firewall for Hadoop cluster,
>> and the client node is outside firewall, I think I have to open some ports
>> that Application Master uses.
>>
>>
>> I think the ports is specified by "spark.yarn.am.port" as document says.
>> https://spark.apache.org/docs/latest/running-on-yarn.html
>>
>> But, according to the source code, spark.yarn.am.port is deprecated since 2.0.
>> https://github.com/apache/spark/commit/829cd7b8b70e65a91aa66e6d626bd45f18e0ad97
>>
>> Does this mean we do not need to open particular ports of firewall for
>>
>> Spark on Yarn?
>>
>>
>> Thanks,
>>
>>
>

Mime
View raw message