spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Or (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-1172) Improve naming of the BlockManager classes
Date Sun, 30 Mar 2014 18:43:14 GMT

    [ https://issues.apache.org/jira/browse/SPARK-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13954789#comment-13954789
] 

Andrew Or commented on SPARK-1172:
----------------------------------

TD and I discussed more about this. I think this issue is part of a broader refactor of the
BlockManager* interface. We should really maintain the abstraction that each actor should
only be for sending/receiving messages, rather than maintaining state, which should be done
in other classes (e.g. BlockManagerMaster).

A bigger issue is that currently the only Akka interface between the driver and the executors
is through the BlockManager* interface. This has not been a problem in the past, until we
started to use this for other things (e.g. cleaning up shuffles in the MapOutputTracker in
the automatic cleanup patch), which becomes quite ugly as BlockManager* classes begin to take
in more and more things as parameters.

> Improve naming of the BlockManager classes
> ------------------------------------------
>
>                 Key: SPARK-1172
>                 URL: https://issues.apache.org/jira/browse/SPARK-1172
>             Project: Apache Spark
>          Issue Type: Improvement
>          Components: Block Manager
>            Reporter: Patrick Wendell
>            Assignee: Andrew Or
>            Priority: Minor
>             Fix For: 1.0.0
>
>
> There is something called BlockManagerMaster that doesn't run on the driver (only on
the Executors). There is a thing called BlockManagerWorker that is not really a "worker" component
but a bridge between the connection manager and the block manager. I found these names confusing
on my first read of this code and in the last week two other people have asked me to explain
it to them. So perhaps we can find more intuitive names here... :)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message