spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pwendell <...@git.apache.org>
Subject [GitHub] spark pull request: [SPARK-1132] Persisting Web UI through refacto...
Date Fri, 14 Mar 2014 07:22:49 GMT
Github user pwendell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/42#discussion_r10598149
  
    --- Diff: core/src/main/scala/org/apache/spark/storage/PutResult.scala ---
    @@ -20,7 +20,13 @@ package org.apache.spark.storage
     import java.nio.ByteBuffer
     
     /**
    - * Result of adding a block into a BlockStore. Contains its estimated size, and possibly
the
    - * values put if the caller asked for them to be returned (e.g. for chaining replication)
    + * Result of adding a block into a BlockStore. This case class contains a few things:
    + *   (1) The estimated size of the put,
    + *   (2) The values put if the caller asked for them to be returned (e.g. for chaining
    + *       replication), and
    + *   (3) A list of blocks dropped as a result of this put. This is always empty for DiskStore.
      */
    -private[spark] case class PutResult(size: Long, data: Either[Iterator[_], ByteBuffer])
    +private[spark] case class PutResult(
    +    size: Long,
    +    data: Either[Iterator[_], ByteBuffer],
    +    droppedBlocks: Seq[(BlockId, BlockStatus)] = Seq())
    --- End diff --
    
    It would be good to add unit tests that test the new logic you've added around these PutResults.
If you look in `BlockManagerSuite` there are fairly extensive tests for the Block Manager.
You should change the existing tests that do puts and cause things to flush out of memory
to verify that the `droppedBlocks` gets updated. And also see if you can add new tests if
those don't sufficiently cover the new code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message