calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julian Hyde <jh...@apache.org>
Subject Re: Using Spark 2.4.0 as Execution Engine with Apache Calcite
Date Wed, 28 Aug 2019 17:28:30 GMT
Your message was received. I guess no one had the time to respond. 

Can you please subscribe to the list? I had to moderate your message through because you are
not a subscriber.

Julian


> On Aug 28, 2019, at 9:03 AM, Shubham Kumar <shubhamkumar1032@gmail.com> wrote:
> 
> Hey,
> 
> Does anyone have any insights on this? Also wanted to confirm that it is
> being sent on the mailing list of Calcite.
> 
> Thanks
> Shubham
> 
> On Mon, Aug 26, 2019 at 6:16 PM Shubham Kumar <shubhamkumar1032@gmail.com>
> wrote:
> 
>> Hey,
>> 
>> I am trying to optimize queries by creating materialized views in Hive and
>> then exposing them to Calcite for optimized query rewrite.
>> 
>> I wanted to know that how mature is the spark adapter of Calcite i.e.
>> Let's say while making the connection I set parameter spark=true
>> <https://calcite.apache.org/apidocs/org/apache/calcite/config/CalciteConnectionProperty.html#SPARK>.
>> Will the workflow be like this?
>> 
>> i) Calcite using Hive's MV's for query rewrites.
>> ii) When query comes ; optimized query rewrite is done by Calcite.
>> iii) This is converted to spark convention and successfully executed using
>> Spark as the engine.
>> 
>> I had a doubt since in many places it's mentioned that spark adapter is
>> not production ready or powerful enough and has been developed for
>> compatibility till spark 1.x only.
>> 
>> Also, can we extract out the query rewrite of Calcite using some API
>> before it is actually executed?
>> --
>> Thanks & Regards
>> Shubham Kumar
>> 
>> 
> 
> -- 
> Thanks & Regards
> 
> Shubham Kumar


Mime
View raw message