spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Mior <mm...@apache.org>
Subject Re: Why Apache Spark doesn't use Calcite?
Date Mon, 13 Jan 2020 16:41:02 GMT
It's fairly common for adapters (Calcite's abstraction of a data
source) to push down predicates. However, the API certainly looks a
lot different than Catalyst's.
--
Michael Mior
mmior@apache.org

Le lun. 13 janv. 2020 à 09:45, Jason Nerothin
<jasonnerothin@gmail.com> a écrit :
>
> The implementation they chose supports push down predicates, Datasets and other features
that are not available in Calcite:
>
> https://databricks.com/glossary/catalyst-optimizer
>
> On Mon, Jan 13, 2020 at 8:24 AM newroyker <newroyker@gmail.com> wrote:
>>
>> Was there a qualitative or quantitative benchmark done before a design
>> decision was made not to use Calcite?
>>
>> Are there limitations (for heuristic based, cost based, * aware optimizer)
>> in Calcite, and frameworks built on top of Calcite? In the context of big
>> data / TCPH benchmarks.
>>
>> I was unable to dig up anything concrete from user group / Jira. Appreciate
>> if any Catalyst veteran here can give me pointers. Trying to defend
>> Spark/Catalyst.
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>
>
> --
> Thanks,
> Jason

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message