spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Leanken.Lin (Jira)" <j...@apache.org>
Subject [jira] [Created] (SPARK-32765) EliminateJoinToEmptyRelation should respect exchange behavior when canChangeNumPartitions == false
Date Tue, 01 Sep 2020 11:56:00 GMT
Leanken.Lin created SPARK-32765:
-----------------------------------

             Summary: EliminateJoinToEmptyRelation should respect exchange behavior when canChangeNumPartitions
== false
                 Key: SPARK-32765
                 URL: https://issues.apache.org/jira/browse/SPARK-32765
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.0.0
         Environment: Currently, EliminateJoinToEmptyRelation Rule will convert Join into
EmptyRelation in some cases with AQE on. But if either sub plan of Join contains a ShuffleQueryStage(canChangeNumPartitions
== false), which means the Exchange produced by repartition Or singlePartition, in this case,
if we were to convert it into an EmptyRelation, it will lost user specified number partition
information for downstream operator, it's not right. 

So in the Patch, try not to do the conversion if either sub plan of Join contains ShuffleQueryStage(canChangeNumPartitions
== false)
            Reporter: Leanken.Lin






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message