spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <>
Subject [jira] [Commented] (SPARK-6951) History server slow startup if the event log directory is large
Date Wed, 01 Mar 2017 11:58:45 GMT


Steve Loughran commented on SPARK-6951:

Having been downstream of YARN timeline server and its leveldb stuff, I'm not convinced adding
a DB will magically solve problems: there's still the need to populate that DB. In a physical
cluster it may stay around, but in a virtual world it won't, and it does complicate life all

With SPARK-17843 giving the UI hints about ongoing async load, at least people can see that
there is a load in progress.

> History server slow startup if the event log directory is large
> ---------------------------------------------------------------
>                 Key: SPARK-6951
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 1.3.0
>            Reporter: Matt Cheah
> I started my history server, then navigated to the web UI where I expected to be able
to view some completed applications, but the webpage was not available. It turned out that
the History Server was not finished parsing all of the event logs in the event log directory
that I had specified. I had accumulated a lot of event logs from months of running Spark,
so it would have taken a very long time for the History Server to crunch through them all.
I purged the event log directory and started from scratch, and the UI loaded immediately.
> We should have a pagination strategy or parse the directory lazily to avoid needing to
wait after starting the history server.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message