spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: InferFiltersFromConstraints logical optimization rule and Optimizer.defaultBatches?
Date Thu, 16 Apr 2020 10:39:20 GMT
Hi Jungtaek,

Thanks a lot for your answer. What you're saying reflects my understanding
perfectly. There's a small change, but makes understanding where rules are
used much simpler (= less confusing). I'll propose a PR and see where it
goes from there. Thanks!

Pozdrawiam,
Jacek Laskowski
----
https://about.me/JacekLaskowski
"The Internals Of" Online Books <https://books.japila.pl/>
Follow me on https://twitter.com/jaceklaskowski

<https://twitter.com/jaceklaskowski>


On Wed, Apr 15, 2020 at 7:55 AM Jungtaek Lim <kabhwan.opensource@gmail.com>
wrote:

> Please correct me if I'm missing something. At a glance, your statements
> look correct if I understand correctly. I guess it might be simply missed,
> but it sounds as pretty trivial one as only a line can be removed safely
> which won't affect anything. (filterNot should be retained even we remove
> the line to prevent extended rules to break this)
>
> On Sun, Apr 12, 2020 at 9:54 PM Jacek Laskowski <jacek@japila.pl> wrote:
>
>> Hi,
>>
>> I'm curious why there is a need to include InferFiltersFromConstraints
>> logical optimization in operatorOptimizationRuleSet value [1] that seems to
>> be only to exclude it later [2]?
>>
>> In other words, I think that simply removing InferFiltersFromConstraints
>> from operatorOptimizationRuleSet value [1] would make no change (except
>> removing a "dead code").
>>
>> Does this make sense? Could I be missing something?
>>
>> [1]
>> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L80
>> [2]
>> https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L115
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://about.me/JacekLaskowski
>> "The Internals Of" Online Books <https://books.japila.pl/>
>> Follow me on https://twitter.com/jaceklaskowski
>>
>> <https://twitter.com/jaceklaskowski>
>>
>

Mime
View raw message