spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <>
Subject [jira] [Commented] (SPARK-20664) Remove stale applications from SHS listing
Date Mon, 22 Jan 2018 21:44:00 GMT


Apache Spark commented on SPARK-20664:

User 'vanzin' has created a pull request for this issue:

> Remove stale applications from SHS listing
> ------------------------------------------
>                 Key: SPARK-20664
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Spark Core
>    Affects Versions: 2.3.0
>            Reporter: Marcelo Vanzin
>            Assignee: Marcelo Vanzin
>            Priority: Major
>             Fix For: 2.3.0
> See spec in parent issue (SPARK-18085) for more details.
> This task is actually not explicit in the spec, and it's also an issue with the current
SHS. But having the SHS persist listing data makes it worse.
> Basically, the SHS currently does not detect when files are deleted from the event log
directory manually; so those applications are still listed, and trying to see their UI will
either show the UI (if it's loaded) or an error (if it's not).
> With the new SHS, that also means that data is leaked in the disk stores used to persist
listing and UI data, making the problem worse.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message