spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Nist <>
Subject Re: Spark Monitoring UI for Hadoop Yarn Cluster
Date Tue, 03 Mar 2015 18:54:17 GMT
Hi Srini,

If you start the $SPARK_HOME/sbin/start-history-server, you should be able
to see the basic spark ui.  You will not see the master, but you will be
able to see the rest as I recall.  You also need to add an entry into the
spark-defaults.conf, something like this:

*## Make sure the host and port match the node where your YARN history
server is running*
spark.yarn.historyServer.address   localhost:18080



On Tue, Mar 3, 2015 at 12:47 PM, Srini Karri <> wrote:

> Hi All,
> I am having trouble finding data related to my requirement. Here is the
> context, I have tried Standalone Spark Installation on Windows, I am able
> to submit the logs, able to see the history of events. My question is, is
> it possible to achieve the same monitoring UI experience with Yarn Cluster
> like Viewing workers, running/completed job stages in the Web UI.
> Currently, if we go to our Yarn Resource manager UI, we are able to see the
> Spark Jobs and it's logs. But it is not as rich as Spark Standalone master
> UI. Is this limitation for hadoop yarn cluster or is there any way we can
> hook this Spark Standalone master to Yarn Cluster?
> Any help is highly appreciated.
> Regards,
> Srini.

View raw message