calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Feng Zhu <wellfeng...@gmail.com>
Subject Re: Using Spark 2.4.0 as Execution Engine with Apache Calcite
Date Thu, 29 Aug 2019 02:43:21 GMT
 Hi,Sgybham,
According to our study, it is developed for Spark 1.x and not powerful in
production.
Rather than relying on Spark adapter, I suggest you convert RelNode back to
SQL based on SparkSqlDialect, and then execute them by Spark SQL.

Shubham Kumar <shubhamkumar1032@gmail.com> 于2019年8月29日周四 上午1:26写道:

> Hey,
>
> Does anyone have any insights on this? Also wanted to confirm that it is
> being sent on the mailing list of Calcite.
>
> Thanks
> Shubham
>
> On Mon, Aug 26, 2019 at 6:16 PM Shubham Kumar <shubhamkumar1032@gmail.com>
> wrote:
>
> > Hey,
> >
> > I am trying to optimize queries by creating materialized views in Hive
> and
> > then exposing them to Calcite for optimized query rewrite.
> >
> > I wanted to know that how mature is the spark adapter of Calcite i.e.
> > Let's say while making the connection I set parameter spark=true
> > <
> https://calcite.apache.org/apidocs/org/apache/calcite/config/CalciteConnectionProperty.html#SPARK
> >.
> > Will the workflow be like this?
> >
> > i) Calcite using Hive's MV's for query rewrites.
> > ii) When query comes ; optimized query rewrite is done by Calcite.
> > iii) This is converted to spark convention and successfully executed
> using
> > Spark as the engine.
> >
> > I had a doubt since in many places it's mentioned that spark adapter is
> > not production ready or powerful enough and has been developed for
> > compatibility till spark 1.x only.
> >
> > Also, can we extract out the query rewrite of Calcite using some API
> > before it is actually executed?
> > --
> > Thanks & Regards
> > Shubham Kumar
> >
> >
>
> --
> Thanks & Regards
>
> Shubham Kumar
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message