spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reynold Xin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-3577) Add task metric to report spill time
Date Mon, 10 Oct 2016 17:41:21 GMT

    [ https://issues.apache.org/jira/browse/SPARK-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15562945#comment-15562945
] 

Reynold Xin commented on SPARK-3577:
------------------------------------

Actually instead of tracking spill time, it's more important to first report spill data size.


> Add task metric to report spill time
> ------------------------------------
>
>                 Key: SPARK-3577
>                 URL: https://issues.apache.org/jira/browse/SPARK-3577
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle, Spark Core
>    Affects Versions: 1.1.0
>            Reporter: Kay Ousterhout
>            Priority: Minor
>
> The {{ExternalSorter}} passes its own {{ShuffleWriteMetrics}} into {{ExternalSorter}}.
 The write time recorded in those metrics is never used.  We should probably add task metrics
to report this spill time, since for shuffles, this would have previously been reported as
part of shuffle write time (with the original hash-based sorter).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message