spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srini Karri <skarri....@gmail.com>
Subject Re: Spark Monitoring UI for Hadoop Yarn Cluster
Date Wed, 04 Mar 2015 19:35:50 GMT
Hi Marcelo,

I found the problem from
http://mail-archives.apache.org/mod_mbox/spark-user/201409.mbox/%3cCAL+LEBfzzjugOoB2iFFdz_=9TQsH=DaiKY=cvydfYDG3Ac5NxA@mail.gmail.com%3e
this link. The problem is the application I am running, is not generating
"APPLICATION_COMPLETE" file. If I add this file manually it is showing
application in the UI. So the problem is with application which is not
calling Stop method on the spark context.

Thank you and Todd for helping. Hopefully I will be able to apply these on
the actual cluster.

Regards,
Srini.

On Wed, Mar 4, 2015 at 10:20 AM, Srini Karri <skarri.net@gmail.com> wrote:

> Yes. I do see files, actually I missed copying the other settings:
>
> spark.master                     spark://
> skarri-lt05.redmond.corp.microsoft.com:7077
> spark.eventLog.enabled           true
> spark.rdd.compress true
> spark.storage.memoryFraction 1
> spark.core.connection.ack.wait.timeout 6000
> spark.akka.frameSize 50
> spark.executor.extraClassPath
> D:\\Apache\\spark-1.2.1-bin-hadoop2\\spark-1.2.1-bin-hadoop2.4\\bin\\classes
> spark.eventLog.dir
> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
> spark.history.fs.logDirectory
> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
>
>
> On Wed, Mar 4, 2015 at 10:15 AM, Marcelo Vanzin <vanzin@cloudera.com>
> wrote:
>
>> On Wed, Mar 4, 2015 at 10:08 AM, Srini Karri <skarri.net@gmail.com>
>> wrote:
>> > spark.executor.extraClassPath
>> >
>> D:\\Apache\\spark-1.2.1-bin-hadoop2\\spark-1.2.1-bin-hadoop2.4\\bin\\classes
>> > spark.eventLog.dir
>> >
>> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
>> > spark.history.fs.logDirectory
>> >
>> D:/Apache/spark-1.2.1-bin-hadoop2/spark-1.2.1-bin-hadoop2.4/bin/tmp/spark-events
>>
>> Do you see any files in that directory?
>>
>> spark.eventLog.dir won't do anything unless you also have
>> spark.eventLog.enabled=true somewhere. And these are application
>> configs, so make sure they're set when running your application (not
>> when starting the history server).
>>
>> --
>> Marcelo
>>
>
>

Mime
View raw message