spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Davis Shepherd (JIRA)" <>
Subject [jira] [Commented] (SPARK-4759) Deadlock in complex spark job in local mode with multiple cores
Date Sun, 07 Dec 2014 20:44:13 GMT


Davis Shepherd commented on SPARK-4759:

It is possible to reproduce the issue with single core local mode.

Simply change the partitions parameter to > 1 (in the attached version it uses the defaultParallelism
of the spark context, which in single core mode is 1) in either of the coalesce calls.

> Deadlock in complex spark job in local mode with multiple cores
> ---------------------------------------------------------------
>                 Key: SPARK-4759
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.1.1, 1.2.0, 1.3.0
>         Environment: Java version "1.7.0_51"
> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
> Mac OSX 10.10.1
> Using local spark context
>            Reporter: Davis Shepherd
>            Assignee: Andrew Or
>            Priority: Critical
>         Attachments: SparkBugReplicator.scala
> The attached test class runs two identical jobs that perform some iterative computation
on an RDD[(Int, Int)]. This computation involves 
>   # taking new data merging it with the previous result
>   # caching and checkpointing the new result
>   # rinse and repeat
> The first time the job is run, it runs successfully, and the spark context is shut down.
The second time the job is run with a new spark context in the same process, the job hangs
indefinitely, only having scheduled a subset of the necessary tasks for the final stage.
> Ive been able to produce a test case that reproduces the issue, and I've added some comments
where some knockout experimentation has left some breadcrumbs as to where the issue might

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message