spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-9487) Use the same num. worker threads in Scala/Python unit tests
Date Sun, 11 Dec 2016 09:25:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15739432#comment-15739432
] 

Sean Owen commented on SPARK-9487:
----------------------------------

I think this is going around in circles. You already have an open invitation to improve tests
in any logical subset of the project in order to accomplish this change in number of worker
threads. You're saying you are unable to get them to pass on Jenkins and unwilling to debug.
I don't think there is more guidance to give here; either you can effect this change or not.
If nobody can or seems willing to try, I think it should be closed, because this really isn't
an error to start with, nor even that suboptimal (excepting that it has revealed a couple
tests could be a little more robust)

> Use the same num. worker threads in Scala/Python unit tests
> -----------------------------------------------------------
>
>                 Key: SPARK-9487
>                 URL: https://issues.apache.org/jira/browse/SPARK-9487
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark, Spark Core, SQL, Tests
>    Affects Versions: 1.5.0
>            Reporter: Xiangrui Meng
>              Labels: starter
>         Attachments: ContextCleanerSuiteResults, HeartbeatReceiverSuiteResults
>
>
> In Python we use `local[4]` for unit tests, while in Scala/Java we use `local[2]` and
`local` for some unit tests in SQL, MLLib, and other components. If the operation depends
on partition IDs, e.g., random number generator, this will lead to different result in Python
and Scala/Java. It would be nice to use the same number in all unit tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message