spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-15974) Create a socket on YARN AM start-up
Date Tue, 12 Feb 2019 21:39:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-15974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Marcelo Vanzin resolved SPARK-15974.
------------------------------------
    Resolution: Won't Fix

The way this feature is proposed, I don't agree with it.

If instead you'd like to expose the user's socket port to YARN, that would be ok, but I think
it would be kinda hard given how the code works.

> Create a socket on YARN AM start-up
> -----------------------------------
>
>                 Key: SPARK-15974
>                 URL: https://issues.apache.org/jira/browse/SPARK-15974
>             Project: Spark
>          Issue Type: New Feature
>          Components: YARN
>            Reporter: Mingyu Kim
>            Priority: Major
>
> YARN provides a way for AppilcationMaster to register a RPC port so that a client outside
the YARN cluster can reach the application for any RPCs, but Spark’s YARN AMs simply register
a dummy port number of 0. For the Spark programs that starts up a server, this makes it hard
for the submitter to discover the server port securely. Spark's ApplicationMaster should optionally
create a ServerSocket and pass it to the Spark user program. This socket initialization should
be disabled by default.
> Some discussion on dev@spark thread: http://apache-spark-developers-list.1001551.n3.nabble.com/Utilizing-YARN-AM-RPC-port-field-td17892.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message