spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (Jira)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-28375) Enforce idempotence on the PullupCorrelatedPredicates optimizer rule
Date Mon, 02 Mar 2020 21:32:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-28375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17049667#comment-17049667
] 

Dongjoon Hyun commented on SPARK-28375:
---------------------------------------

Although we don't backport this, I updated this issue type as `Bug` and updated the affected
version due to `postHocOptimizationBatches`.

> Enforce idempotence on the PullupCorrelatedPredicates optimizer rule
> --------------------------------------------------------------------
>
>                 Key: SPARK-28375
>                 URL: https://issues.apache.org/jira/browse/SPARK-28375
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0, 2.3.0, 2.4.0, 3.0.0
>            Reporter: Yesheng Ma
>            Assignee: Dilip Biswal
>            Priority: Major
>              Labels: correctness
>             Fix For: 3.0.0
>
>
> The current PullupCorrelatedPredicates implementation can accidentally remove predicates
for multiple runs.
> For example, for the following logical plan, one more optimizer run can remove the predicate
in the SubqueryExpresssion.
> {code:java}
> # Optimized
> Project [a#0]
> +- Filter a#0 IN (list#4 [(b#1 < d#3)])
>    :  +- Project [c#2, d#3]
>    :     +- LocalRelation <empty>, [c#2, d#3]
>    +- LocalRelation <empty>, [a#0, b#1]
> # Double optimized
> Project [a#0]
> +- Filter a#0 IN (list#4 [])
>    :  +- Project [c#2, d#3]
>    :     +- LocalRelation <empty>, [c#2, d#3]
>    +- LocalRelation <empty>, [a#0, b#1]
> {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message