spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-28536) Reduce shuffle partitions in Python UDF tests in SQLQueryTestSuite
Date Sat, 27 Jul 2019 17:48:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-28536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dongjoon Hyun resolved SPARK-28536.
-----------------------------------
       Resolution: Fixed
         Assignee: Hyukjin Kwon
    Fix Version/s: 3.0.0

This is resolved via https://github.com/apache/spark/pull/25271

> Reduce shuffle partitions in Python UDF tests in SQLQueryTestSuite
> ------------------------------------------------------------------
>
>                 Key: SPARK-28536
>                 URL: https://issues.apache.org/jira/browse/SPARK-28536
>             Project: Spark
>          Issue Type: Test
>          Components: PySpark, SQL
>    Affects Versions: 3.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>             Fix For: 3.0.0
>
>
> Currently, some SQL tests with Python UDFs takes long.
> In my local:
> {code:java}
> [info] SQLQueryTestSuite:
> [info] - udf/udf-window.sql - Scala UDF (58 seconds, 558 milliseconds)
> [info] - udf/udf-window.sql - Regular Python UDF (58 seconds, 371 milliseconds)
> [info] - udf/udf-window.sql - Scalar Pandas UDF (1 minute, 8 seconds){code}
> and it takes up to 9 mins in Jenkins currently.
> In Python UDF tests, the number of shuffle partitions matter considerably in testing
time because it requires to fork and communicate between external processes. We should reduce
the number of it.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message