spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maxim Gekk (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-25286) Remove dangerous parmap
Date Thu, 30 Aug 2018 20:56:00 GMT
Maxim Gekk created SPARK-25286:
----------------------------------

             Summary: Remove dangerous parmap
                 Key: SPARK-25286
                 URL: https://issues.apache.org/jira/browse/SPARK-25286
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 2.4.0
            Reporter: Maxim Gekk


One of parmap methods accepts an execution context created outside of parmap. If the parmap
method is called recursively on a thread pool limited by size, it could lead to deadlocks.
See the JIRA tickets: SPARK-25240 and SPARK-25283 . To eliminate the problems in the future,
need to remove parmap() with the signature:
{code:scala}
def parmap[I, O, Col[X] <: TraversableLike[X, Col[X]]]
      (in: Col[I])
      (f: I => O)
      (implicit
        cbf: CanBuildFrom[Col[I], Future[O], Col[Future[O]]], // For in.map
        cbf2: CanBuildFrom[Col[Future[O]], O, Col[O]], // for Future.sequence
        ec: ExecutionContext
      ): Col[O]
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message