spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jiang Xingbo (JIRA)" <>
Subject [jira] [Commented] (SPARK-24817) Implement BarrierTaskContext.barrier()
Date Thu, 02 Aug 2018 02:54:00 GMT


Jiang Xingbo commented on SPARK-24817:

Actually the current implementation of _barrier_ function doesn't requires communications
between executors, all executors just talk to a _BarrierCoordinator_ which is in the driver.
But to allow launching ML workloads we do need to enable executors to communicate with each
other directly, IIUC that shall be investigated under SPARK-24724 . Maybe [~mengxr] can provide
more context here.

> Implement BarrierTaskContext.barrier()
> --------------------------------------
>                 Key: SPARK-24817
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 2.4.0
>            Reporter: Jiang Xingbo
>            Priority: Major
> Implement BarrierTaskContext.barrier(), to support global sync between all the tasks
in a barrier stage. The global sync shall finish immediately once all tasks in the same barrier
stage reaches the same barrier.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message