spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maxim Gekk (JIRA)" <>
Subject [jira] [Created] (SPARK-25286) Remove dangerous parmap
Date Thu, 30 Aug 2018 20:56:00 GMT
Maxim Gekk created SPARK-25286:

             Summary: Remove dangerous parmap
                 Key: SPARK-25286
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 2.4.0
            Reporter: Maxim Gekk

One of parmap methods accepts an execution context created outside of parmap. If the parmap
method is called recursively on a thread pool limited by size, it could lead to deadlocks.
See the JIRA tickets: SPARK-25240 and SPARK-25283 . To eliminate the problems in the future,
need to remove parmap() with the signature:
def parmap[I, O, Col[X] <: TraversableLike[X, Col[X]]]
      (in: Col[I])
      (f: I => O)
        cbf: CanBuildFrom[Col[I], Future[O], Col[Future[O]]], // For
        cbf2: CanBuildFrom[Col[Future[O]], O, Col[O]], // for Future.sequence
        ec: ExecutionContext
      ): Col[O]

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message