spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erik Erlandson (JIRA)" <>
Subject [jira] [Commented] (SPARK-24817) Implement BarrierTaskContext.barrier()
Date Wed, 01 Aug 2018 23:46:00 GMT


Erik Erlandson commented on SPARK-24817:

I'm curious about what the {{barrier}} invocations inside {{mapPartitions}} closures imply
about communications between executors, for example executors running on pods in a kube cluster.
It is possible that whatever is allowing shuffle data to transfer between executors will also
allow these  {{barrier}} coordinations to work, but we had to create a headless service
for executors to register properly with the driver pod, and if every executor pod needs something
like that for barrier to work, it will be an impact for kube backend support.

> Implement BarrierTaskContext.barrier()
> --------------------------------------
>                 Key: SPARK-24817
>                 URL:
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 2.4.0
>            Reporter: Jiang Xingbo
>            Priority: Major
> Implement BarrierTaskContext.barrier(), to support global sync between all the tasks
in a barrier stage. The global sync shall finish immediately once all tasks in the same barrier
stage reaches the same barrier.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message