spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vishwas Nalka (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-29465) Unable to configure SPARK UI (spark.ui.port) in spark yarn cluster mode.
Date Wed, 16 Oct 2019 05:36:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-29465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952516#comment-16952516
] 

Vishwas Nalka edited comment on SPARK-29465 at 10/16/19 5:35 AM:
-----------------------------------------------------------------

[~dongjoon], The problem is when the yarn cluster is running in a machine where not all ports
can be opened. The requirement is to restrict the ports used by the spark job that is launched
in yarn mode. I was able to set all other ports like _"spark.driver.port", "spark.blockManager.port"_
except the UI port.
{code:java}
// Set the web ui port to be ephemeral for yarn so we don't conflict with
// other spark processes running on the same box
System.setProperty("spark.ui.port", "0"){code}

 Can't the above code be modified to include a condition to check if the UI port is already
set by the user, if not the port should be set to random as mentioned in the comments.  Do
let me know your suggestion.

Thanks!


was (Author: vishwasn):
[~dongjoon], The problem is when the yarn cluster is running in a machine where not all ports
can be opened. The requirement is to restrict the ports used by the spark job that is launched
in yarn mode. I was able to set all other ports like _"spark.driver.port", "spark.blockManager.port"_
except the UI port.
// Set the web ui port to be ephemeral for yarn so we don't conflict with// other spark processes
running on the same boxSystem.setProperty("spark.ui.port", "0")
Can't the above code be modified to include a condition to check if the UI port is already
set by the user, if not the port should be set to random as mentioned in the comments.  Do
let me know your suggestion.

Thanks!

> Unable to configure SPARK UI (spark.ui.port) in spark yarn cluster mode. 
> -------------------------------------------------------------------------
>
>                 Key: SPARK-29465
>                 URL: https://issues.apache.org/jira/browse/SPARK-29465
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Submit, YARN
>    Affects Versions: 2.4.4
>            Reporter: Vishwas Nalka
>            Priority: Major
>
>  I'm trying to restrict the ports used by spark app which is launched in yarn cluster
mode. All ports (viz. driver, executor, blockmanager) could be specified using the respective
properties except the ui port. The spark app is launched using JAVA code and setting the property
spark.ui.port in sparkConf doesn't seem to help. Even setting a JVM option -Dspark.ui.port="some_port"
does not spawn the UI is required port. 
> From the logs of the spark app, *_the property spark.ui.port is overridden and the JVM
property '-Dspark.ui.port=0' is set_* even though it is never set to 0. 
> _(Run in Spark 1.6.2) From the logs ->_
> _command:LD_LIBRARY_PATH="/usr/hdp/2.6.4.0-91/hadoop/lib/native:$LD_LIBRARY_PATH" {{JAVA_HOME}}/bin/java
-server -XX:OnOutOfMemoryError='kill %p' -Xms4096m -Xmx4096m -Djava.io.tmpdir={{PWD}}/tmp
'-Dspark.blockManager.port=9900' '-Dspark.driver.port=9902' '-Dspark.fileserver.port=9903'
'-Dspark.broadcast.port=9904' '-Dspark.port.maxRetries=20' '-Dspark.ui.port=0' '-Dspark.executor.port=9905'_
> _19/10/14 16:39:59 INFO Utils: Successfully started service 'SparkUI' on port 35167.19/10/14
16:39:59 INFO SparkUI: Started SparkUI at_ [_http://10.65.170.98:35167_|http://10.65.170.98:35167/]
> Even tried using a *spark-submit command with --conf spark.ui.port* does spawn UI in
required port
> {color:#172b4d}_(Run in Spark 2.4.4)_{color}
>  {color:#172b4d}_./bin/spark-submit --class org.apache.spark.examples.SparkPi --master
yarn --deploy-mode cluster --driver-memory 4g --executor-memory 2g --executor-cores 1 --conf
spark.ui.port=12345 --conf spark.driver.port=12340 --queue default examples/jars/spark-examples_2.11-2.4.4.jar
10_{color}
> _From the logs::_
>  _19/10/15 00:04:05 INFO ui.SparkUI: Stopped Spark web UI at [http://invrh74ace005.informatica.com:46622|http://invrh74ace005.informatica.com:46622/]_
> _command:{{JAVA_HOME}}/bin/java -server -Xmx2048m -Djava.io.tmpdir={{PWD}}/tmp '-Dspark.ui.port=0' 
'Dspark.driver.port=12340' -Dspark.yarn.app.container.log.dir=<LOG_DIR> -XX:OnOutOfMemoryError='kill
%p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@invrh74ace005.informatica.com:12340
--executor-id <executorId> --hostname <hostname> --cores 1 --app-id application_1570992022035_0089
--user-class-path [file:$PWD/__app__.jar1|file://%24pwd/__app__.jar1]><LOG_DIR>/stdout2><LOG_DIR>/stderr_
>  
> Looks like the application master override this and set a JVM property before launch
resulting in random UI port even though spark.ui.port is set by the user.
> In these links
>  # [https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala]
(line 214)
>  # [https://github.com/cloudera/spark/blob/master/yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala]
(line 75)
> I can see that the method _*run() in above files sets a system property UI_PORT*_ and _*spark.ui.port
respectively.*_



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message