spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saikat Kanjilal (JIRA)" <>
Subject [jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests
Date Sat, 10 Dec 2016 20:56:58 GMT


Saikat Kanjilal commented on SPARK-9487:

Then I would suggest keeping it open and focus on a particular module and make the unit tests
robust in that module, is there a specific module that's in dire need of robustness of unit
tests, I was thinking of picking the sql module and moving forward to make the unit tests
under that be more robust as a first goal, thoughts?

> Use the same num. worker threads in Scala/Python unit tests
> -----------------------------------------------------------
>                 Key: SPARK-9487
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark, Spark Core, SQL, Tests
>    Affects Versions: 1.5.0
>            Reporter: Xiangrui Meng
>              Labels: starter
>         Attachments: ContextCleanerSuiteResults, HeartbeatReceiverSuiteResults
> In Python we use `local[4]` for unit tests, while in Scala/Java we use `local[2]` and
`local` for some unit tests in SQL, MLLib, and other components. If the operation depends
on partition IDs, e.g., random number generator, this will lead to different result in Python
and Scala/Java. It would be nice to use the same number in all unit tests.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message