spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Unit tests in < 5 minutes
Date Fri, 05 Dec 2014 00:33:22 GMT
Have you seen this thread http://search-hadoop.com/m/JW1q5xxSAa2 ?

Test categorization in HBase is done through maven-surefire-plugin

Cheers

On Thu, Dec 4, 2014 at 4:05 PM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:

> fwiw, when we did this work in HBase, we categorized the tests. Then some
> tests can share a single jvm, while some others need to be isolated in
> their own jvm. Nevertheless surefire can still run them in parallel by
> starting/stopping several jvm.
>
> I think we need to do this as well. Perhaps the test naming hierarchy can
> be used to group non-parallelizable tests in the same JVM.
>
> For example, here are some Hive tests from our project:
>
> org.apache.spark.sql.hive.StatisticsSuite
> org.apache.spark.sql.hive.execution.HiveQuerySuite
> org.apache.spark.sql.QueryTest
> org.apache.spark.sql.parquet.HiveParquetSuite
>
> If we group tests by the first 5 parts of their name (e.g.
> org.apache.spark.sql.hive), then we’d have the first 2 tests run in the
> same JVM, and the next 2 tests each run in their own JVM.
>
> I’m new to this stuff so I’m not sure if I’m going about this in the right
> way, but you can see my attempt with this approach on GitHub
> <https://github.com/nchammas/spark/blob/ab127b798dbfa9399833d546e627f9651b060918/project/SparkBuild.scala#L388-L397>,
> as well as the related discussion on JIRA
> <https://issues.apache.org/jira/browse/SPARK-3431>.
>
> If anyone has more feedback on this, I’d love to hear it (either on this
> thread or in the JIRA issue).
>
> Nick
> ​
>
> On Sun Sep 07 2014 at 8:28:51 PM Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> On Fri, Aug 8, 2014 at 1:12 PM, Reynold Xin <rxin@databricks.com> wrote:
>>
>>> Nick,
>>>
>>> Would you like to file a ticket to track this?
>>>
>>
>> SPARK-3431 <https://issues.apache.org/jira/browse/SPARK-3431>:
>> Parallelize execution of tests
>> > Sub-task: SPARK-3432 <https://issues.apache.org/jira/browse/SPARK-3432>:
>> Fix logging of unit test execution time
>>
>> Nick
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message