spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sven Krasser (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-4049) Storage web UI "fraction cached" shows as > 100%
Date Thu, 29 Jan 2015 00:08:34 GMT

    [ https://issues.apache.org/jira/browse/SPARK-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296037#comment-14296037
] 

Sven Krasser edited comment on SPARK-4049 at 1/29/15 12:07 AM:
---------------------------------------------------------------

-I'm also seeing this for a 2x replicated RDD (StorageLevel.MEMORY_AND_DISK_2). I assume that
means that most partitions are replicated twice and some three times?- EDIT: looks like for
2x it will ramp up to 200%, which is consistent with Patrick's comment.

That aside, I do not think it's a good idea to count overreplication towards that fraction
in this way. As a user, when I see 100% on the UI, then I assume the RDD is fully cached.
However, this could also mean that some partitions are missing (and need to be recomputed)
and some are overreplicated.


was (Author: skrasser):
I'm also seeing this for a 2x replicated RDD (StorageLevel.MEMORY_AND_DISK_2). I assume that
means that most partitions are replicated twice and some three times?

That aside, I do not think it's a good idea to count overreplication towards that fraction
in this way. As a user, when I see 100% on the UI, then I assume the RDD is fully cached.
However, this could also mean that some partitions are missing (and need to be recomputed)
and some are overreplicated.

> Storage web UI "fraction cached" shows as > 100%
> ------------------------------------------------
>
>                 Key: SPARK-4049
>                 URL: https://issues.apache.org/jira/browse/SPARK-4049
>             Project: Spark
>          Issue Type: Bug
>          Components: Web UI
>    Affects Versions: 1.2.0
>            Reporter: Josh Rosen
>            Priority: Minor
>
> In the Storage tab of the Spark Web UI, I saw a case where the "Fraction Cached" was
greater than 100%:
> !http://i.imgur.com/Gm2hEeL.png!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message