Hi Grzegorz:
      I have a similar  scenario with you, but even I called the sc.stop(), there is no  APPLICATION_COMPLETE  file in the log directory. can you share some experiment for this problem. Thanks very much.  

On Mon, Sep 15, 2014 at 4:10 PM, Grzegorz Białek <grzegorz.bialek@codilime.com> wrote:
Hi Andrew,

sorry for late response. Thank you very much for solving my problem. There was no APPLICATION_COMPLETE file in log directory due to not calling sc.stop() at the end of program. With stopping spark context everything works correctly, so thank you again.

Best regards,

On Fri, Sep 5, 2014 at 8:06 PM, Andrew Or <andrew@databricks.com> wrote:
Hi Grzegorz,

Can you verify that there are "APPLICATION_COMPLETE" files in the event log directories? E.g. Does file:/tmp/spark-events/app-name-1234567890/APPLICATION_COMPLETE exist? If not, it could be that your application didn't call sc.stop(), so the "ApplicationEnd" event is not actually logged. The HistoryServer looks for this special file to identify applications to display. You could also try manually adding the "APPLICATION_COMPLETE" file to this directory; the HistoryServer should pick this up and display the application, though the information displayed will be incomplete because the log did not capture all the events (sc.stop() does a final close() on the file written).


2014-09-05 1:50 GMT-07:00 Grzegorz Białek <grzegorz.bialek@codilime.com>:

Hi Andrew,

thank you very much for your answer. Unfortunately it still doesn't work. I'm using Spark 1.0.0, and I start history server running sbin/start-history-server.sh <dir>, although I also set SPARK_HISTORY_OPTS=-Dspark.history.fs.logDirectory in conf/spark-env.sh. I tried also other dir than /tmp/spark-events which have all possible permissions enabled. Also adding file: (and file://) didn't help - history server still shows:
History Server
Event Log Location: file:/tmp/spark-events/
No Completed Applications Found.

Best regards,

On Thu, Sep 4, 2014 at 8:20 PM, Andrew Or <andrew@databricks.com> wrote:
Hi Grzegorz,

Sorry for the late response. Unfortunately, if the Master UI doesn't know about your applications (they are "completed" with respect to a different Master), then it can't regenerate the UIs even if the logs exist. You will have to use the history server for that.

How did you start the history server? If you are using Spark <=1.0, you can pass the directory as an argument to the sbin/start-history-server.sh script. Otherwise, you may need to set the following in your conf/spark-env.sh to specify the log directory:

export SPARK_HISTORY_OPTS=-Dspark.history.fs.logDirectory=/tmp/spark-events

It could also be a permissions thing. Make sure your logs in /tmp/spark-events are accessible by the JVM that runs the history server. Also, there's a chance that "/tmp/spark-events" is interpreted as an HDFS path depending on which Spark version you're running. To resolve any ambiguity, you may set the log path to "file:/tmp/spark-events" instead. But first verify whether they actually exist.

Let me know if you get it working,

2014-08-19 8:23 GMT-07:00 Grzegorz Białek <grzegorz.bialek@codilime.com>:

Is there any way view history of applications statistics in master ui after restarting master server? I have all logs ing /tmp/spark-events/ but when I start history server in this directory it says "No Completed Applications Found". Maybe I could copy this logs to dir used by master server but I couldn't find any. Or maybe I'm doing something wrong launching history server.
Do you have any idea how to solve it?


On Thu, Aug 14, 2014 at 10:53 AM, Grzegorz Białek <grzegorz.bialek@codilime.com> wrote:

Thank you both for your answers. Browsing using Master UI works fine. Unfortunately History Server shows "No Completed Applications Found" even if logs exists under given directory, but using Master UI is enough for me.

Best regards,

On Wed, Aug 13, 2014 at 8:09 PM, Andrew Or <andrew@databricks.com> wrote:
The Spark UI isn't available through the same address; otherwise new applications won't be able to bind to it. Once the old application finishes, the standalone Master renders the after-the-fact application UI and exposes it under a different URL. To see this, go to the Master UI (<master-url>:8080) and click on your application in the "Completed Applications" table.

2014-08-13 10:56 GMT-07:00 Matei Zaharia <matei.zaharia@gmail.com>:

Take a look at http://spark.apache.org/docs/latest/monitoring.html -- you need to launch a history server to serve the logs.


On August 13, 2014 at 2:03:08 AM, grzegorz-bialek (grzegorz.bialek@codilime.com) wrote:

I wanted to access Spark web UI after application stops. I set
spark.eventLog.enabled to true and logs are availaible
in JSON format in /tmp/spark-event but web UI isn't available under address
I'm running Spark in standalone mode.

What should I do to access web UI after application ends?


View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Viewing-web-UI-after-fact-tp12023.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org