spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-15076) Improve ConstantFolding optimizer to use integral associative property
Date Mon, 02 May 2016 22:38:12 GMT
Dongjoon Hyun created SPARK-15076:
-------------------------------------

             Summary: Improve ConstantFolding optimizer to use integral associative property
                 Key: SPARK-15076
                 URL: https://issues.apache.org/jira/browse/SPARK-15076
             Project: Spark
          Issue Type: Improvement
          Components: Optimizer
            Reporter: Dongjoon Hyun


This issue improves `ConstantFolding` optimizer by taking advantage of integral associative
property. Currently, `ConstantFolding` works like the following.

1) Can optimize `1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9 + a` into `45 + a`.
2) Cannot optimize `a + 1 + 2 + 3 + 4 + 5 + 6 + 7 + 8 + 9`.

This issue enables `ConstantFolding` to handle Case 2 for *Add/Multiply* expression whose
data types are `ByteType`, `ShortType`, `IntegerType`, and `LongType`. The followings are
the plan comparision between `before` and `after` this issue.

**Before**
{code}
scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1)) a)").explain
== Physical Plan ==
WholeStageCodegen
:  +- Project [(((((((((a#7 + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9) AS (((((((((a + 1)
+ 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9)#8]
:     +- INPUT
+- Generate explode([1]), false, false, [a#7]
   +- Scan OneRowRelation[]
{code}

**After**
{code}
scala> sql("select a+1+2+3+4+5+6+7+8+9 from (select explode(array(1)) a)").explain
== Physical Plan ==
WholeStageCodegen
:  +- Project [(a#7 + 45) AS (((((((((a + 1) + 2) + 3) + 4) + 5) + 6) + 7) + 8) + 9)#8]
:     +- INPUT
+- Generate explode([1]), false, false, [a#7]
   +- Scan OneRowRelation[]
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message