calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shubham Kumar <>
Subject Re: Using Spark 2.4.0 as Execution Engine with Apache Calcite
Date Wed, 28 Aug 2019 16:03:24 GMT

Does anyone have any insights on this? Also wanted to confirm that it is
being sent on the mailing list of Calcite.


On Mon, Aug 26, 2019 at 6:16 PM Shubham Kumar <>

> Hey,
> I am trying to optimize queries by creating materialized views in Hive and
> then exposing them to Calcite for optimized query rewrite.
> I wanted to know that how mature is the spark adapter of Calcite i.e.
> Let's say while making the connection I set parameter spark=true
> <>.
> Will the workflow be like this?
> i) Calcite using Hive's MV's for query rewrites.
> ii) When query comes ; optimized query rewrite is done by Calcite.
> iii) This is converted to spark convention and successfully executed using
> Spark as the engine.
> I had a doubt since in many places it's mentioned that spark adapter is
> not production ready or powerful enough and has been developed for
> compatibility till spark 1.x only.
> Also, can we extract out the query rewrite of Calcite using some API
> before it is actually executed?
> --
> Thanks & Regards
> Shubham Kumar

Thanks & Regards

Shubham Kumar

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message