spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nicholas Chammas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-3431) Parallelize execution of tests
Date Wed, 10 Dec 2014 02:01:13 GMT

    [ https://issues.apache.org/jira/browse/SPARK-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14240519#comment-14240519
] 

Nicholas Chammas commented on SPARK-3431:
-----------------------------------------

OK, thanks for the updates, Sean and Nicolas.

On my side, I've gone back to testing with SBT to better understand what's going wrong there.
Specifically, why the [working directory appears to be different|https://issues.apache.org/jira/browse/SPARK-3431?focusedCommentId=14236540&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14236540]
when when we fork the JVM or not.

I came across [this SBT issue|https://github.com/sbt/sbt/issues/1032] which seems to document
a known behavior of SBT in multiproject builds. Forking vs. not forking does appear to change
the working directory, which I can confirm is what broke the HiveQuerySuite test with {{java.io.IOException:
Cannot run program "/usr/bin/hadoop"}}.

> Parallelize execution of tests
> ------------------------------
>
>                 Key: SPARK-3431
>                 URL: https://issues.apache.org/jira/browse/SPARK-3431
>             Project: Spark
>          Issue Type: Improvement
>          Components: Build
>            Reporter: Nicholas Chammas
>            Assignee: Nicholas Chammas
>         Attachments: SPARK-3431-srowen-attempt.patch
>
>
> Running all the tests in {{dev/run-tests}} takes up to 2 hours. A common strategy to
cut test time down is to parallelize the execution of the tests. Doing that may in turn require
some prerequisite changes to be made to how certain tests run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message