spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srini Karri <skarri....@gmail.com>
Subject Re: Spark Monitoring UI for Hadoop Yarn Cluster
Date Wed, 04 Mar 2015 18:20:59 GMT
Yes. I do see files, actually I missed copying the other settings:

spark.master                     spark://
skarri-lt05.redmond.corp.microsoft.com:7077
spark.eventLog.enabled           true
spark.rdd.compress true
spark.storage.memoryFraction 1
spark.core.connection.ack.wait.timeout 6000
spark.akka.frameSize 50
spark.executor.extraClassPath
D:\\Apache\\spark-1.2.1-bin-hadoop2\\spark-1.2.1-bin-hadoop2.4\\bin\\classes
spark.eventLog.dir
D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
spark.history.fs.logDirectory
D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events


On Wed, Mar 4, 2015 at 10:15 AM, Marcelo Vanzin <vanzin@cloudera.com> wrote:

> On Wed, Mar 4, 2015 at 10:08 AM, Srini Karri <skarri.net@gmail.com> wrote:
> > spark.executor.extraClassPath
> >
> D:\\Apache\\spark-1.2.1-bin-hadoop2\\spark-1.2.1-bin-hadoop2.4\\bin\\classes
> > spark.eventLog.dir
> >
> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
> > spark.history.fs.logDirectory
> >
> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
>
> Do you see any files in that directory?
>
> spark.eventLog.dir won't do anything unless you also have
> spark.eventLog.enabled=true somewhere. And these are application
> configs, so make sure they're set when running your application (not
> when starting the history server).
>
> --
> Marcelo
>

Mime
View raw message