spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (JIRA)" <>
Subject [jira] [Assigned] (SPARK-28535) Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader"
Date Sat, 27 Jul 2019 18:08:00 GMT


Dongjoon Hyun reassigned SPARK-28535:

    Assignee: Marcelo Vanzin

> Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader"
> ---------------------------------------------------------------------------
>                 Key: SPARK-28535
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Tests
>    Affects Versions: 2.3.3, 3.0.0, 2.4.3
>            Reporter: Marcelo Vanzin
>            Assignee: Marcelo Vanzin
>            Priority: Minor
> This is the same flakiness as in SPARK-23881, except the fix there didn't really take,
at least on our build machines.
> {noformat}
> org.scalatest.exceptions.TestFailedException: 10000 was not less than 10000
>       at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:528)
>       at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
>       at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
> {noformat}
> Since that bug is short on explanations, the issue is that there's a race between the
thread posting the "stage completed" event to the listener which unblocks the test, and the
thread killing the task in the executor. If the even arrives first, it will unblock task execution,
and there's a chance that all elements will actually be processed before the executor has
a chance to stop the task.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message