spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jungtaek Lim <kabhwan.opensou...@gmail.com>
Subject Re: InferFiltersFromConstraints logical optimization rule and Optimizer.defaultBatches?
Date Wed, 15 Apr 2020 05:55:45 GMT
Please correct me if I'm missing something. At a glance, your statements
look correct if I understand correctly. I guess it might be simply missed,
but it sounds as pretty trivial one as only a line can be removed safely
which won't affect anything. (filterNot should be retained even we remove
the line to prevent extended rules to break this)

On Sun, Apr 12, 2020 at 9:54 PM Jacek Laskowski <jacek@japila.pl> wrote:

> Hi,
>
> I'm curious why there is a need to include InferFiltersFromConstraints
> logical optimization in operatorOptimizationRuleSet value [1] that seems to
> be only to exclude it later [2]?
>
> In other words, I think that simply removing InferFiltersFromConstraints
> from operatorOptimizationRuleSet value [1] would make no change (except
> removing a "dead code").
>
> Does this make sense? Could I be missing something?
>
> [1]
> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L80
> [2]
> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L115
>
> Pozdrawiam,
> Jacek Laskowski
> ----
> https://about.me/JacekLaskowski
> "The Internals Of" Online Books <https://books.japila.pl/>
> Follow me on https://twitter.com/jaceklaskowski
>
> <https://twitter.com/jaceklaskowski>
>

Mime
View raw message