spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Richard Marscher <rmarsc...@localytics.com>
Subject Re: spark eventLog and history server
Date Tue, 09 Jun 2015 17:52:31 GMT
Hi,

I don't have a complete answer to your questions but:

"Removing the suffix does not solve the problem" -> unfortunately this is
true, the master web UI only tries to build out a Spark UI from the event
logs once, at the time the context is closed. If the event logs are
in-progress at this time, then you basically missed the opportunity.

"Does it mean I don't need to start history server if I only use spark in
standalone mode?" -> Yes, you don't need to start the history server.

On Mon, Jun 8, 2015 at 7:57 PM, Du Li <lidu@yahoo-inc.com.invalid> wrote:

> Event log is enabled in my spark streaming app. My code runs in standalone
> mode and the spark version is 1.3.1. I periodically stop and restart the
> streaming context by calling ssc.stop(). However, from the web UI, when
> clicking on a past job, it says the job is still in progress and does not
> show the event log. The event log files have suffix .inprogress. Removing
> the suffix does not solve the problem. Do I need to do anything here in
> order to view the event logs of finished jobs? Or do I need to stop ssc
> differently?
>
> In addition, the documentation seems to suggest history server is used for
> Mesos or YARN mode. Does it mean I don't need to start history server if I
> only use spark in standalone mode?
>
> Thanks,
> Du
>

Mime
View raw message