spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tomer Benyamini <tomer....@gmail.com>
Subject Re: Spark UI port issue when deploying Spark driver on YARN in yarn-cluster mode on EMR
Date Tue, 23 Dec 2014 17:18:39 GMT
On YARN, spark does not manage the cluster, but YARN does. Usually the
cluster manager UI is under http://<yarn master>:9026/cluster. I believe
that it chooses the port for the spark driver UI randomly, but an easy way
of accessing it is by clicking on the "Application Master" link under the
"Tracking UI" column in the cluster manager UI. Note that by default on
EMR, this link will use the amazon internal ip, so you need to set up a vpn
to view these kind of links from your browser.

Tomer

On Tue, Dec 23, 2014 at 6:04 PM, Roberto Coluccio <
roberto.coluccio@gmail.com> wrote:

> Hello folks,
>
> I'm trying to deploy a Spark driver on Amazon EMR in yarn-cluster mode
> expecting to be able to access the Spark UI from the <spark-master-ip>:4040
> address (default port). The problem here is that the Spark UI port is
> always defined randomly at runtime, although I also tried to specify it in
> the spark-defaults.conf file: in order to do so, I used this:
> https://github.com/awslabs/emr-bootstrap-actions/tree/master/spark#3-utilize-an-emr-step-to-configure-the-spark-default-configuration-optional
> , setting the spark.ui.port to a static known value. No luck, every time I
> launch (using the spark-submit script from the yarn-master node) a Spark
> driver, the UI port is chose randomly.
>
> Is there any configurations I'm missing out here?
>
> Thank you very much.
>
> Roberto
>

Mime
View raw message