calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stamatis Zampetakis <zabe...@gmail.com>
Subject Re: ClassCastException RelOptCostImpl VolcanoCost while using HepPlanner
Date Wed, 08 May 2019 13:37:29 GMT
In order to use the planner you can instantiate directly without
necessarily passing by Frameworks.
For an end to end example on how to parse, validate, plan, execute a query
using the basic primitives provided by Calcite have a look in [1].

[1]
https://github.com/zabetak/calcite/blob/livecodingdemo/core/src/test/java/org/apache/calcite/examples/foodmart/java/EndToEndExampleEnumerable.java

On Wed, May 8, 2019 at 9:41 AM Mark Pasterkamp <markpasterkamp1994@gmail.com>
wrote:

> I don't necessarily want to pass relnodes from one planner to another, but
> I see no other way how to do it.
> From my understanding, if you have a sql string you need a planner to parse
> and validate it into a relnode.
> The PlannerImpl created by Frameworks.getPlanner() has an empty internal
> planner which gets initialized via the Frameworks.withPlanner function[1].
>
> Frameworks.withPlanner then subsequently calls the Frameworks.withPrepare
> [2] which calls the CalcitePrepareImpl.perform() [3].
> There the planner gets created using the createPlanner method [4]. It is
> there that the planner is finally created as a VolcanoPlanner [5].
> Maybe it can be considered to make this planner configurable in one of the
> config settings instead of it being a hardcoded VolcanoPlanner.
>
> The one thing I can still try is to use the RelOptCostImpl.Factory [6] to
> make the VolcanoPlanner return RelOptCostImpl instead of VolcanoCost.
>
> Or perhaps there is another way how I can parse a string into RelNodes that
> I do not know about, then I would love to know.
>
> [1]
>
> https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/prepare/PlannerImpl.java#L137
> [2]
>
> https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/tools/Frameworks.java#L114
> [3]
>
> https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/tools/Frameworks.java#L157
> [4]
>
> https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L1038
> [5]
>
> https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L520
> [6]
>
> https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/plan/RelOptCostImpl.java#L122
>
>
>
> On Tue, 7 May 2019 at 13:21, Stamatis Zampetakis <zabetak@gmail.com>
> wrote:
>
> > Thanks for looking more into this Mark.
> >
> > Passing RelNodes from one planner to another is not a good idea.
> > Ideally this should be possible but I guess it requires CALCITE-1536 [1]
> to
> > be solved.
> >
> > Another way to avoid problems like the one you observed is to copy
> RelNodes
> > from one cluster to another [2].
> >
> > Best,
> > Stamatis
> >
> > [1] https://issues.apache.org/jira/browse/CALCITE-1536
> > [2] https://issues.apache.org/jira/browse/CALCITE-1681
> >
> >
> > On Mon, May 6, 2019 at 12:02 PM Mark Pasterkamp <
> > markpasterkamp1994@gmail.com> wrote:
> >
> > > I think I found the origin of the exception.
> > > To convert a single query to a rel node I am using the PlannerImp
> > provided
> > > by Frameworks.getPlanner().
> > > This planner provides a VolcanoPlanner to the relnode clusters.
> > > When calling the HepPlanner.findBestExp() it will eventually call
> > > the applyTransformationResults which will compute a cost using the
> > > rel.getCluster().getPlanner() (which is a VolcanoPlanner).
> > >
> > >  Mark
> > >
> > > On Fri, 3 May 2019 at 19:56, Mark Pasterkamp <
> > markpasterkamp1994@gmail.com
> > > >
> > > wrote:
> > >
> > > > I did not know this is how it works, I just copied the example above.
> > > > Would there be an easy way to create a RelNode containg a tablescan
> > over
> > > > the materialized view "mv"?
> > > > Trying to create one using for instance a relbuilder gives a calcite
> > > > exception.
> > > > Otherwise I might just look for another test file in which I can get
> > > > access to the schema and use the materiaqlizationservice.
> > > >
> > > > Mark
> > > >
> > > > On Fri, 3 May 2019 at 18:35, Jesus Camacho Rodriguez
> > > > <jcamachorodriguez@cloudera.com.invalid> wrote:
> > > >
> > > >> bq . Since we are talking about materialized views, I think in most
> > > cases
> > > >> tableRel should be simply a LogicalTableScan.
> > > >>
> > > >> Stamatis is correct about this, I had not realized  tableRel ==
> > queryRel
> > > >> in
> > > >> your sample code.
> > > >>
> > > >> Thanks,
> > > >> Jesús
> > > >>
> > > >> On Fri, May 3, 2019 at 6:12 AM Stamatis Zampetakis <
> zabetak@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >> > I think the main problem comes from the fact that tableRel ==
> > queryRel
> > > >> in
> > > >> > the test case you provided.
> > > >> > Defining the materialized view like that basically says that
when
> > you
> > > >> find
> > > >> > a part of the query that satisfies queryRel replace it with
> itself.
> > > >> > In conjunction with the rule that is used, which allows partial
> > > >> rewritings
> > > >> > using union, you end up with a rule that matches infinite number
> of
> > > >> times.
> > > >> > Since we are talking about materialized views, I think in most
> cases
> > > >> > tableRel should be simply a LogicalTableScan.
> > > >> > The idea is that expression represented by queryRel is
> materialized
> > > >> into a
> > > >> > table so in order to retrieve the results we only need to scan
the
> > > >> table.
> > > >> >
> > > >> > Regarding the "if (true)" statements that you observed, most
> likely
> > > they
> > > >> > were introduced as release toggles [1].
> > > >> > However, since the last commit was in 2013 I think by now it
is
> safe
> > > to
> > > >> > refactor that part and remove dead code.
> > > >> >
> > > >> > [1] https://www.martinfowler.com/articles/feature-toggles.html
> > > >> >
> > > >> > Best,
> > > >> > Stamatis
> > > >> >
> > > >> > On Fri, May 3, 2019 at 12:50 PM Mark Pasterkamp <
> > > >> > markpasterkamp1994@gmail.com> wrote:
> > > >> >
> > > >> > > Dear Jesus,
> > > >> > >
> > > >> > > I think your intuition in this regard is correct.
> > > >> > > After executing the main program in the HepPlanner the resulting
> > > plan
> > > >> > > contains a lot of circular references.
> > > >> > > Changing the matching order does not influence this behaviour.
> > > >> > >
> > > >> > >
> > > >> > > Mark
> > > >> > >
> > > >> > > On Thu, 2 May 2019 at 22:14, Jesus Camacho Rodriguez
> > > >> > > <jcamachorodriguez@cloudera.com.invalid> wrote:
> > > >> > >
> > > >> > > > Mark,
> > > >> > > >
> > > >> > > > I have an intuition that this happens because the rule
> creates a
> > > >> > > partially
> > > >> > > > contained rewriting with a union, where one side contains
a
> scan
> > > >> over
> > > >> > the
> > > >> > > > materialized view and the other side contains the query
itself
> > > with
> > > >> a
> > > >> > > > filter on top excluding the data that is coming from
the
> > > >> materialized
> > > >> > > view.
> > > >> > > > Then the rule is triggered on the plan representing
the
> original
> > > >> query
> > > >> > > > again and the process is repeated. Have you tried changing
the
> > > >> matching
> > > >> > > > order for your hep program?
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Jesús
> > > >> > > >
> > > >> > > > On Thu, May 2, 2019 at 8:53 AM Mark Pasterkamp <
> > > >> > > > markpasterkamp1994@gmail.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi Stamatis,
> > > >> > > > >
> > > >> > > > > I have tried to recreate the issue but I have
not been able
> to
> > > do
> > > >> > > that. I
> > > >> > > > > was however able to create a new exception which
I don't
> quite
> > > >> > > > understand.
> > > >> > > > > The error happened when calcite was creating a
union
> rewriting
> > > >> using
> > > >> > > > > materialized views. But trying to recreate this
situation
> gave
> > > me
> > > >> > > another
> > > >> > > > > interesting one.
> > > >> > > > > This time, the planner rewrites one of the children
nodes
> into
> > > >> > itself I
> > > >> > > > > would assume which causes a stack overflow. The
method
> itself
> > > can
> > > >> be
> > > >> > > > found
> > > >> > > > > here:
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/mpasterkamp/calcite/blob/768b7928dbde5f6f9775a1119e7466d8eafafb4b/core/src/test/java/org/apache/calcite/test/HepPlannerTest.java#L312
> > > >> > > > >
> > > >> > > > > Perhaps I am doing something wrong, perhaps not?
I am not
> > > >> > knowledgeable
> > > >> > > > > enough about this to understand why this is happening.
Wish
> I
> > > >> could
> > > >> > > help
> > > >> > > > > more for that.
> > > >> > > > >
> > > >> > > > > Also, while investigating this issue I found another
> > interesting
> > > >> > > artifact
> > > >> > > > > in de source code of the VolcanoCost. A lot of
methods in
> this
> > > >> class
> > > >> > > have
> > > >> > > > > an "if (true)"-statement like here:
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/calcite/blob/4b4d8037c5073e4eb5702b12bc4ecade31476616/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoCost.java#L100
> > > >> > > > >
> > > >> > > > > Now I was just curious, is there any reason for
this to be
> > there
> > > >> that
> > > >> > > you
> > > >> > > > > know of?
> > > >> > > > >
> > > >> > > > > Thank you for responding and congratulations for
your recent
> > > >> > > promotions.
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > With kind regards,
> > > >> > > > >
> > > >> > > > > Mark
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Thu, 2 May 2019 at 14:58, Stamatis Zampetakis
<
> > > >> zabetak@gmail.com>
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Said like that it looks like a bug.
> > > >> > > > > >
> > > >> > > > > > I think the best would be to reproduce the
exception as a
> > unit
> > > >> test
> > > >> > > in
> > > >> > > > > > HepPlannerTest [1], RelOptRulesTest [2],
or PlannerTest
> [3]
> > so
> > > >> that
> > > >> > > we
> > > >> > > > > > could understand better the use case.
> > > >> > > > > >
> > > >> > > > > > [1]
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/HepPlannerTest.java
> > > >> > > > > > [2]
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java
> > > >> > > > > > [3]
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/tools/PlannerTest.java
> > > >> > > > > >
> > > >> > > > > > On Thu, May 2, 2019 at 7:56 AM Mark Pasterkamp
<
> > > >> > > > > > markpasterkamp1994@gmail.com>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > I don't, I would assume that the
> HepPlanner.findBestExp()
> > > >> > > calculates
> > > >> > > > > the
> > > >> > > > > > > cost somewhere down the line
> > > >> > > > > > >
> > > >> > > > > > > On Thu, May 2, 2019, 03:31 Yuzhao Chen
<
> > > yuzhao.cyz@gmail.com>
> > > >> > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Why you care about cost when use
HepPlanner ? The
> > > >> HepPlanner is
> > > >> > > > aimed
> > > >> > > > > > for
> > > >> > > > > > > > some deterministic planning rules,
we usually do not
> > need
> > > >> cost
> > > >> > in
> > > >> > > > > Hep.
> > > >> > > > > > > Some
> > > >> > > > > > > > exceptions like Join reorder may
need a cost.
> > > >> > > > > > > >
> > > >> > > > > > > > What kind of planning promotion
you did ? I'm kind of
> > > >> curious
> > > >> > > about
> > > >> > > > > it.
> > > >> > > > > > > >
> > > >> > > > > > > > Best,
> > > >> > > > > > > > Danny Chan
> > > >> > > > > > > > 在 2019年5月1日 +0800 PM9:27,Mark
Pasterkamp <
> > > >> > > > > markpasterkamp1994@gmail.com
> > > >> > > > > > > > >,写道:
> > > >> > > > > > > > > Dear all,
> > > >> > > > > > > > >
> > > >> > > > > > > > > While playing around with
the HepPlanner I ran into
> an
> > > >> issue
> > > >> > > > where
> > > >> > > > > > the
> > > >> > > > > > > > > planner wants to rewrite a
query with a union
> rewrite.
> > > >> When
> > > >> > the
> > > >> > > > > > > > > RelMetaDataQuery computes
the cost, the cost
> instance
> > > is a
> > > >> > > > > > VolcanoCost.
> > > >> > > > > > > > > Then when it tries to calculate
the cost of one of
> the
> > > >> > union's
> > > >> > > > > > operands
> > > >> > > > > > > > it
> > > >> > > > > > > > > is a RelCostImpl which results
in the
> > > ClassCastException.
> > > >> > > > > > > > >
> > > >> > > > > > > > > How would I go about solving
this issue? As far as
> my
> > > >> > knowledge
> > > >> > > > > > goes, I
> > > >> > > > > > > > am
> > > >> > > > > > > > > not able to change the costhandler
of the
> > > >> RelMetaDataQuery.
> > > >> > > > Another
> > > >> > > > > > > > > approach I could see is removing
the cast in the
> > > >> VolcanoCost
> > > >> > > > class,
> > > >> > > > > > > but I
> > > >> > > > > > > > > would hope I do not have to
do that.
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > With kind regards,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Mark
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message