spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "deshanxiao (Jira)" <j...@apache.org>
Subject [jira] [Created] (SPARK-30123) PartitionPruning should consider more case
Date Wed, 04 Dec 2019 13:17:00 GMT
deshanxiao created SPARK-30123:
----------------------------------

             Summary: PartitionPruning should consider more case
                 Key: SPARK-30123
                 URL: https://issues.apache.org/jira/browse/SPARK-30123
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.0.0
            Reporter: deshanxiao


If left has partitionScan and right has PruningFilter but hasBenefit is false. The right will
never be added a SubQuery.


{code:java}
            var partScan = getPartitionTableScan(l, left)
            if (partScan.isDefined && canPruneLeft(joinType) &&
                hasPartitionPruningFilter(right)) {
              val hasBenefit = pruningHasBenefit(l, partScan.get, r, right)
              newLeft = insertPredicate(l, newLeft, r, right, rightKeys, hasBenefit)
            } else {
              partScan = getPartitionTableScan(r, right)
              if (partScan.isDefined && canPruneRight(joinType) &&
                  hasPartitionPruningFilter(left) ) {
                val hasBenefit = pruningHasBenefit(r, partScan.get, l, left)
                newRight = insertPredicate(r, newRight, l, left, leftKeys, hasBenefit)
              }
            }
          case _ =>
        }
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message