calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shubham Kumar <shubhamkumar1...@gmail.com>
Subject Using Spark 2.4.0 as Execution Engine with Apache Calcite
Date Mon, 26 Aug 2019 12:46:08 GMT
Hey,

I am trying to optimize queries by creating materialized views in Hive and
then exposing them to Calcite for optimized query rewrite.

I wanted to know that how mature is the spark adapter of Calcite i.e. Let's
say while making the connection I set parameter spark=true
<https://calcite.apache.org/apidocs/org/apache/calcite/config/CalciteConnectionProperty.html#SPARK>.
Will the workflow be like this?

i) Calcite using Hive's MV's for query rewrites.
ii) When query comes ; optimized query rewrite is done by Calcite.
iii) This is converted to spark convention and successfully executed using
Spark as the engine.

I had a doubt since in many places it's mentioned that spark adapter is not
production ready or powerful enough and has been developed for
compatibility till spark 1.x only.

Also, can we extract out the query rewrite of Calcite using some API before
it is actually executed?
-- 
Thanks & Regards
Shubham Kumar

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message