calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Pasterkamp <markpasterkamp1...@gmail.com>
Subject Re: ClassCastException RelOptCostImpl VolcanoCost while using HepPlanner
Date Wed, 08 May 2019 07:41:00 GMT
I don't necessarily want to pass relnodes from one planner to another, but
I see no other way how to do it.
>From my understanding, if you have a sql string you need a planner to parse
and validate it into a relnode.
The PlannerImpl created by Frameworks.getPlanner() has an empty internal
planner which gets initialized via the Frameworks.withPlanner function[1].

Frameworks.withPlanner then subsequently calls the Frameworks.withPrepare
[2] which calls the CalcitePrepareImpl.perform() [3].
There the planner gets created using the createPlanner method [4]. It is
there that the planner is finally created as a VolcanoPlanner [5].
Maybe it can be considered to make this planner configurable in one of the
config settings instead of it being a hardcoded VolcanoPlanner.

The one thing I can still try is to use the RelOptCostImpl.Factory [6] to
make the VolcanoPlanner return RelOptCostImpl instead of VolcanoCost.

Or perhaps there is another way how I can parse a string into RelNodes that
I do not know about, then I would love to know.

[1]
https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/prepare/PlannerImpl.java#L137
[2]
https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/tools/Frameworks.java#L114
[3]
https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/tools/Frameworks.java#L157
[4]
https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L1038
[5]
https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L520
[6]
https://github.com/apache/calcite/blob/f5363478e1feed6a090b2ac0c9fc52743a653ca0/core/src/main/java/org/apache/calcite/plan/RelOptCostImpl.java#L122



On Tue, 7 May 2019 at 13:21, Stamatis Zampetakis <zabetak@gmail.com> wrote:

> Thanks for looking more into this Mark.
>
> Passing RelNodes from one planner to another is not a good idea.
> Ideally this should be possible but I guess it requires CALCITE-1536 [1] to
> be solved.
>
> Another way to avoid problems like the one you observed is to copy RelNodes
> from one cluster to another [2].
>
> Best,
> Stamatis
>
> [1] https://issues.apache.org/jira/browse/CALCITE-1536
> [2] https://issues.apache.org/jira/browse/CALCITE-1681
>
>
> On Mon, May 6, 2019 at 12:02 PM Mark Pasterkamp <
> markpasterkamp1994@gmail.com> wrote:
>
> > I think I found the origin of the exception.
> > To convert a single query to a rel node I am using the PlannerImp
> provided
> > by Frameworks.getPlanner().
> > This planner provides a VolcanoPlanner to the relnode clusters.
> > When calling the HepPlanner.findBestExp() it will eventually call
> > the applyTransformationResults which will compute a cost using the
> > rel.getCluster().getPlanner() (which is a VolcanoPlanner).
> >
> >  Mark
> >
> > On Fri, 3 May 2019 at 19:56, Mark Pasterkamp <
> markpasterkamp1994@gmail.com
> > >
> > wrote:
> >
> > > I did not know this is how it works, I just copied the example above.
> > > Would there be an easy way to create a RelNode containg a tablescan
> over
> > > the materialized view "mv"?
> > > Trying to create one using for instance a relbuilder gives a calcite
> > > exception.
> > > Otherwise I might just look for another test file in which I can get
> > > access to the schema and use the materiaqlizationservice.
> > >
> > > Mark
> > >
> > > On Fri, 3 May 2019 at 18:35, Jesus Camacho Rodriguez
> > > <jcamachorodriguez@cloudera.com.invalid> wrote:
> > >
> > >> bq . Since we are talking about materialized views, I think in most
> > cases
> > >> tableRel should be simply a LogicalTableScan.
> > >>
> > >> Stamatis is correct about this, I had not realized  tableRel ==
> queryRel
> > >> in
> > >> your sample code.
> > >>
> > >> Thanks,
> > >> Jesús
> > >>
> > >> On Fri, May 3, 2019 at 6:12 AM Stamatis Zampetakis <zabetak@gmail.com
> >
> > >> wrote:
> > >>
> > >> > I think the main problem comes from the fact that tableRel ==
> queryRel
> > >> in
> > >> > the test case you provided.
> > >> > Defining the materialized view like that basically says that when
> you
> > >> find
> > >> > a part of the query that satisfies queryRel replace it with itself.
> > >> > In conjunction with the rule that is used, which allows partial
> > >> rewritings
> > >> > using union, you end up with a rule that matches infinite number of
> > >> times.
> > >> > Since we are talking about materialized views, I think in most cases
> > >> > tableRel should be simply a LogicalTableScan.
> > >> > The idea is that expression represented by queryRel is materialized
> > >> into a
> > >> > table so in order to retrieve the results we only need to scan the
> > >> table.
> > >> >
> > >> > Regarding the "if (true)" statements that you observed, most likely
> > they
> > >> > were introduced as release toggles [1].
> > >> > However, since the last commit was in 2013 I think by now it is safe
> > to
> > >> > refactor that part and remove dead code.
> > >> >
> > >> > [1] https://www.martinfowler.com/articles/feature-toggles.html
> > >> >
> > >> > Best,
> > >> > Stamatis
> > >> >
> > >> > On Fri, May 3, 2019 at 12:50 PM Mark Pasterkamp <
> > >> > markpasterkamp1994@gmail.com> wrote:
> > >> >
> > >> > > Dear Jesus,
> > >> > >
> > >> > > I think your intuition in this regard is correct.
> > >> > > After executing the main program in the HepPlanner the resulting
> > plan
> > >> > > contains a lot of circular references.
> > >> > > Changing the matching order does not influence this behaviour.
> > >> > >
> > >> > >
> > >> > > Mark
> > >> > >
> > >> > > On Thu, 2 May 2019 at 22:14, Jesus Camacho Rodriguez
> > >> > > <jcamachorodriguez@cloudera.com.invalid> wrote:
> > >> > >
> > >> > > > Mark,
> > >> > > >
> > >> > > > I have an intuition that this happens because the rule creates
a
> > >> > > partially
> > >> > > > contained rewriting with a union, where one side contains
a scan
> > >> over
> > >> > the
> > >> > > > materialized view and the other side contains the query
itself
> > with
> > >> a
> > >> > > > filter on top excluding the data that is coming from the
> > >> materialized
> > >> > > view.
> > >> > > > Then the rule is triggered on the plan representing the
original
> > >> query
> > >> > > > again and the process is repeated. Have you tried changing
the
> > >> matching
> > >> > > > order for your hep program?
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Jesús
> > >> > > >
> > >> > > > On Thu, May 2, 2019 at 8:53 AM Mark Pasterkamp <
> > >> > > > markpasterkamp1994@gmail.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi Stamatis,
> > >> > > > >
> > >> > > > > I have tried to recreate the issue but I have not been
able to
> > do
> > >> > > that. I
> > >> > > > > was however able to create a new exception which I
don't quite
> > >> > > > understand.
> > >> > > > > The error happened when calcite was creating a union
rewriting
> > >> using
> > >> > > > > materialized views. But trying to recreate this situation
gave
> > me
> > >> > > another
> > >> > > > > interesting one.
> > >> > > > > This time, the planner rewrites one of the children
nodes into
> > >> > itself I
> > >> > > > > would assume which causes a stack overflow. The method
itself
> > can
> > >> be
> > >> > > > found
> > >> > > > > here:
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/mpasterkamp/calcite/blob/768b7928dbde5f6f9775a1119e7466d8eafafb4b/core/src/test/java/org/apache/calcite/test/HepPlannerTest.java#L312
> > >> > > > >
> > >> > > > > Perhaps I am doing something wrong, perhaps not? I
am not
> > >> > knowledgeable
> > >> > > > > enough about this to understand why this is happening.
Wish I
> > >> could
> > >> > > help
> > >> > > > > more for that.
> > >> > > > >
> > >> > > > > Also, while investigating this issue I found another
> interesting
> > >> > > artifact
> > >> > > > > in de source code of the VolcanoCost. A lot of methods
in this
> > >> class
> > >> > > have
> > >> > > > > an "if (true)"-statement like here:
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/calcite/blob/4b4d8037c5073e4eb5702b12bc4ecade31476616/core/src/main/java/org/apache/calcite/plan/volcano/VolcanoCost.java#L100
> > >> > > > >
> > >> > > > > Now I was just curious, is there any reason for this
to be
> there
> > >> that
> > >> > > you
> > >> > > > > know of?
> > >> > > > >
> > >> > > > > Thank you for responding and congratulations for your
recent
> > >> > > promotions.
> > >> > > > >
> > >> > > > >
> > >> > > > > With kind regards,
> > >> > > > >
> > >> > > > > Mark
> > >> > > > >
> > >> > > > >
> > >> > > > > On Thu, 2 May 2019 at 14:58, Stamatis Zampetakis <
> > >> zabetak@gmail.com>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Said like that it looks like a bug.
> > >> > > > > >
> > >> > > > > > I think the best would be to reproduce the exception
as a
> unit
> > >> test
> > >> > > in
> > >> > > > > > HepPlannerTest [1], RelOptRulesTest [2], or PlannerTest
[3]
> so
> > >> that
> > >> > > we
> > >> > > > > > could understand better the use case.
> > >> > > > > >
> > >> > > > > > [1]
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/HepPlannerTest.java
> > >> > > > > > [2]
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/test/RelOptRulesTest.java
> > >> > > > > > [3]
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/calcite/blob/master/core/src/test/java/org/apache/calcite/tools/PlannerTest.java
> > >> > > > > >
> > >> > > > > > On Thu, May 2, 2019 at 7:56 AM Mark Pasterkamp
<
> > >> > > > > > markpasterkamp1994@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > I don't, I would assume that the HepPlanner.findBestExp()
> > >> > > calculates
> > >> > > > > the
> > >> > > > > > > cost somewhere down the line
> > >> > > > > > >
> > >> > > > > > > On Thu, May 2, 2019, 03:31 Yuzhao Chen <
> > yuzhao.cyz@gmail.com>
> > >> > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Why you care about cost when use HepPlanner
? The
> > >> HepPlanner is
> > >> > > > aimed
> > >> > > > > > for
> > >> > > > > > > > some deterministic planning rules, we
usually do not
> need
> > >> cost
> > >> > in
> > >> > > > > Hep.
> > >> > > > > > > Some
> > >> > > > > > > > exceptions like Join reorder may need
a cost.
> > >> > > > > > > >
> > >> > > > > > > > What kind of planning promotion you
did ? I'm kind of
> > >> curious
> > >> > > about
> > >> > > > > it.
> > >> > > > > > > >
> > >> > > > > > > > Best,
> > >> > > > > > > > Danny Chan
> > >> > > > > > > > 在 2019年5月1日 +0800 PM9:27,Mark
Pasterkamp <
> > >> > > > > markpasterkamp1994@gmail.com
> > >> > > > > > > > >,写道:
> > >> > > > > > > > > Dear all,
> > >> > > > > > > > >
> > >> > > > > > > > > While playing around with the HepPlanner
I ran into an
> > >> issue
> > >> > > > where
> > >> > > > > > the
> > >> > > > > > > > > planner wants to rewrite a query
with a union rewrite.
> > >> When
> > >> > the
> > >> > > > > > > > > RelMetaDataQuery computes the cost,
the cost instance
> > is a
> > >> > > > > > VolcanoCost.
> > >> > > > > > > > > Then when it tries to calculate
the cost of one of the
> > >> > union's
> > >> > > > > > operands
> > >> > > > > > > > it
> > >> > > > > > > > > is a RelCostImpl which results
in the
> > ClassCastException.
> > >> > > > > > > > >
> > >> > > > > > > > > How would I go about solving this
issue? As far as my
> > >> > knowledge
> > >> > > > > > goes, I
> > >> > > > > > > > am
> > >> > > > > > > > > not able to change the costhandler
of the
> > >> RelMetaDataQuery.
> > >> > > > Another
> > >> > > > > > > > > approach I could see is removing
the cast in the
> > >> VolcanoCost
> > >> > > > class,
> > >> > > > > > > but I
> > >> > > > > > > > > would hope I do not have to do
that.
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > With kind regards,
> > >> > > > > > > > >
> > >> > > > > > > > > Mark
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message