spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (Jira)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-29164) Rewrite coalesce(boolean, booleanLit) as boolean expression
Date Tue, 17 Dec 2019 02:05:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-29164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hyukjin Kwon resolved SPARK-29164.
----------------------------------
    Resolution: Won't Fix

> Rewrite coalesce(boolean, booleanLit) as boolean expression
> -----------------------------------------------------------
>
>                 Key: SPARK-29164
>                 URL: https://issues.apache.org/jira/browse/SPARK-29164
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: Josh Rosen
>            Priority: Major
>
> I propose the following expression rewrite optimizations:
> {code:java}
> coalesce(x: Boolean, true)  -> x or isnull(x)
> coalesce(x: Boolean, false) -> x and isnotnull(x){code}
> This pattern appears when translating Dataset filters on {{Option[Boolean]}} columns:
we might have a typed Dataset filter which looks like
> {code:java}
>  .filter(_.boolCol.getOrElse(DEFAULT_VALUE)){code}
> and the most idiomatic, user-friendly translation of this in Catalyst is to use {{coalesce()}}.
However, the {{coalesce()}} form of this expression is not eligible for Parquet / data source
filter pushdown.
> (We should write out truth-tables to double-check this rewrite's correctness)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message