spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reynold Xin (JIRA)" <>
Subject [jira] [Resolved] (SPARK-15353) Making peer selection for block replication pluggable
Date Sat, 01 Oct 2016 01:25:20 GMT


Reynold Xin resolved SPARK-15353.
       Resolution: Fixed
         Assignee: Shubham Chopra
    Fix Version/s: 2.1.0

> Making peer selection for block replication pluggable
> -----------------------------------------------------
>                 Key: SPARK-15353
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Block Manager, Spark Core
>            Reporter: Shubham Chopra
>            Assignee: Shubham Chopra
>             Fix For: 2.1.0
>         Attachments: BlockManagerSequenceDiagram.png
> BlockManagers running on executors provide all logistics around block management. Before
a BlockManager can be used, it has to be “initialized”. As a part of the initialization,
BlockManager asks the BlockManagerMasterEndpoint to give it topology information. The BlockManagerMasterEndpoint
is provided a pluggable interface that can be used to resolve a hostname to topology. This
information is used to decorate the BlockManagerId. This happens at cluster start and whenever
a new executor is added.
> During replication, the BlockManager gets the list of all its peers in the form of a
Seq[BlockManagerId]. We add a pluggable prioritizer that can be used to prioritize this list
of peers based on topology information. Peers with higher priority occur first in the sequence
and the BlockManager tries to replicate blocks in that order.
> There would be default implementations for these pluggable interfaces that replicate
the existing behavior of randomly choosing a peer.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message