spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grzegorz Białek <grzegorz.bia...@codilime.com>
Subject Re: Viewing web UI after fact
Date Fri, 05 Sep 2014 08:50:41 GMT
Hi Andrew,

thank you very much for your answer. Unfortunately it still doesn't work.
I'm using Spark 1.0.0, and I start history server running
sbin/start-history-server.sh <dir>, although I also set
 SPARK_HISTORY_OPTS=-Dspark.history.fs.logDirectory in conf/spark-env.sh. I
tried also other dir than /tmp/spark-events which have all possible
permissions enabled. Also adding file: (and file://) didn't help - history
server still shows:
History Server
Event Log Location: file:/tmp/spark-events/
No Completed Applications Found.

Best regards,
Grzegorz


On Thu, Sep 4, 2014 at 8:20 PM, Andrew Or <andrew@databricks.com> wrote:

> Hi Grzegorz,
>
> Sorry for the late response. Unfortunately, if the Master UI doesn't know
> about your applications (they are "completed" with respect to a different
> Master), then it can't regenerate the UIs even if the logs exist. You will
> have to use the history server for that.
>
> How did you start the history server? If you are using Spark <=1.0, you
> can pass the directory as an argument to the sbin/start-history-server.sh
> script. Otherwise, you may need to set the following in your
> conf/spark-env.sh to specify the log directory:
>
> export SPARK_HISTORY_OPTS=-Dspark.history.fs.logDirectory=/tmp/spark-events
>
> It could also be a permissions thing. Make sure your logs in
> /tmp/spark-events are accessible by the JVM that runs the history server.
> Also, there's a chance that "/tmp/spark-events" is interpreted as an HDFS
> path depending on which Spark version you're running. To resolve any
> ambiguity, you may set the log path to "file:/tmp/spark-events" instead.
> But first verify whether they actually exist.
>
> Let me know if you get it working,
> -Andrew
>
>
>
> 2014-08-19 8:23 GMT-07:00 Grzegorz Białek <grzegorz.bialek@codilime.com>:
>
> Hi,
>> Is there any way view history of applications statistics in master ui
>> after restarting master server? I have all logs ing /tmp/spark-events/ but
>> when I start history server in this directory it says "No Completed
>> Applications Found". Maybe I could copy this logs to dir used by master
>> server but I couldn't find any. Or maybe I'm doing something wrong
>> launching history server.
>> Do you have any idea how to solve it?
>>
>> Thanks,
>> Grzegorz
>>
>>
>> On Thu, Aug 14, 2014 at 10:53 AM, Grzegorz Białek <
>> grzegorz.bialek@codilime.com> wrote:
>>
>>> Hi,
>>>
>>> Thank you both for your answers. Browsing using Master UI works fine.
>>> Unfortunately History Server shows "No Completed Applications Found" even
>>> if logs exists under given directory, but using Master UI is enough for me.
>>>
>>> Best regards,
>>> Grzegorz
>>>
>>>
>>>
>>> On Wed, Aug 13, 2014 at 8:09 PM, Andrew Or <andrew@databricks.com>
>>> wrote:
>>>
>>>> The Spark UI isn't available through the same address; otherwise new
>>>> applications won't be able to bind to it. Once the old application
>>>> finishes, the standalone Master renders the after-the-fact application UI
>>>> and exposes it under a different URL. To see this, go to the Master UI
>>>> (<master-url>:8080) and click on your application in the "Completed
>>>> Applications" table.
>>>>
>>>>
>>>> 2014-08-13 10:56 GMT-07:00 Matei Zaharia <matei.zaharia@gmail.com>:
>>>>
>>>> Take a look at http://spark.apache.org/docs/latest/monitoring.html --
>>>>> you need to launch a history server to serve the logs.
>>>>>
>>>>> Matei
>>>>>
>>>>> On August 13, 2014 at 2:03:08 AM, grzegorz-bialek (
>>>>> grzegorz.bialek@codilime.com) wrote:
>>>>>
>>>>> Hi,
>>>>> I wanted to access Spark web UI after application stops. I set
>>>>> spark.eventLog.enabled to true and logs are availaible
>>>>> in JSON format in /tmp/spark-event but web UI isn't available under
>>>>> address
>>>>> http://<driver-node>:4040
>>>>> I'm running Spark in standalone mode.
>>>>>
>>>>> What should I do to access web UI after application ends?
>>>>>
>>>>> Thanks,
>>>>> Grzegorz
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Viewing-web-UI-after-fact-tp12023.html
>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>> Nabble.com.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message