ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Seliverstov Igor <gvvinbl...@gmail.com>
Subject Re: New SQL execution engine
Date Fri, 27 Sep 2019 16:03:04 GMT
Nikolay,

> What project hosted Calcite based engine?


Currently the prototype is placed in my personal Ignite fork. I need an appropriate ticket
before pushing it to ASF git repository. 
At first, I think, we should discuss the idea in general.

> Personally, I'm against the support of two independent implementation of SQL engine for
several releases.


I don’t like the idea to have two engines too. But even development the engine on top of
Calcite library is still a big deal. 
I not sure it will be ready, no, I sure it WONT be ready by Ignite3 release. So I mentioned
the option to have two engines at the same time.

> Let's start with the IEP clarification and replace the SQL engine with the best one for
Ignite good.

Of course, but anyway it’s good to make familiar with a couple of examples it already describes
and clarify some additional questions the community may ask.

Regards,
Igor

> 27 сент. 2019 г., в 18:22, Nikolay Izhikov <nizhikov@apache.org> написал(а):
> 
> Igor.
> 
>> There is no decision, here we should decide.
> 
> Great.
> 
>> At now Calcite based engine is placed in different module
> 
> What project hosted Calcite based engine?
> 
>> It’s possible to develop it as an experimental extension at first (not a replacement)
> 
> For me, Ignite 3 are the place where the new engine has to be placed.
> Personally, I'm against the support of two independent implementation of SQL engine for
several releases.
> 
> Ignite has too many partially implemented features to include on more :)
> 
> Let's start with the IEP clarification and replace the SQL engine with the best one for
Ignite good.
> 
> 
> В Пт, 27/09/2019 в 18:08 +0300, Seliverstov Igor пишет:
>> Nikolay,
>> 
>> At last we have better questions.
>> 
>> There is no decision, here we should decide.
>> 
>> Doing nothing isn’t a decision, it’s just doing nothing
>> 
>> Spark Catalyst is a good example, but under the hood it has absolutely the same idea,
but adopted to Spark. Calcite is the same, but general. That’s why it’s better start point.
>> 
>> Implementing an engine from scratch is really cool, but looks like inventing a bicycle,
don’t think it makes sense. At least I against this option.
>> 
>> I added requirements to IEP (as you asked), you may see it’s in DRAFT state and
will be complemented by details.
>> 
>> We have some thoughts on how to make smooth replacement, but at first we should decide
what to replace and what with.
>> 
>> At now Calcite based engine is placed in different module, we checked it can build
execution graph for both local and distributed cases, it has good expandability. 
>> We talked to Calcite community to identify possible future issues and everything
points to the fact it’s the best option. 
>> It’s possible to develop it as an experimental extension at first (not a replacement)
until we make sure that it works as expected. This way there are no risks for anybody who
uses Ignite on production environment.
>> 
>> Regards,
>> Igor
>> 
>> 
>>> 27 сент. 2019 г., в 17:25, Nikolay Izhikov <nizhikov@apache.org>
написал(а):
>>> 
>>> Igor.
>>> 
>>>> The main issue - there is no *selection*.
>>> 
>>> 1. I don't remember community decision about this.
>>> 
>>> 2. We should avoid to make such long-term decision so quickly.
>>> We done this kind of decision with H2 and come to the point when we should review
it.
>>> 
>>>> 1) Implementing white papers from scratch
>>>> 2) Adopting Calcite to our needs.
>>> 
>>> The third option don't fix issues we have with H2.
>>> The fourth option I know is using spark-catalyst.
>>> 
>>> What is wrong with writing engine from scratch?
>>> 
>>> I ask you to start with engine requirements.
>>> Can we, please, discuss it?
>>> 
>>>> If you have an alternative - you're welcome, I'll gratefully listen to you.
>>> 
>>> We have alternative for now - H2 based engine.
>>> 
>>>> The main question isn't "WHAT" but "HOW" - that's the discussion topic from
my point of view.
>>> 
>>> When we make a decision about engine we can discuss roadmap for replacement.
>>> One more time - replacement of SQL engine to some more customizable make sense
for me.
>>> But, this kind of decisions need carefull discussion.
>>> 
>>> В Пт, 27/09/2019 в 17:08 +0300, Seliverstov Igor пишет:
>>>> Nikolay,
>>>> 
>>>> The main issue - there is no *selection*.
>>>> 
>>>> There is a field of knowledge - relational algebra, which describes how to
transform relational expressions saving their semantics, and a couple of implementations (Calcite
is only one written in Java).
>>>> 
>>>> There are only two alternatives:
>>>> 
>>>> 1) Implementing white papers from scratch
>>>> 2) Adopting Calcite to our needs.
>>>> 
>>>> The second way was chosen by several other projects, there is experience,
there is a list of known issues (like using indexes) so, almost everything is already done
for us.
>>>> 
>>>> Implementing a planner is a big deal, I think anybody understands it there.
That's why our proposal to reuse others experience is obvious.
>>>> 
>>>> If you have an alternative - you're welcome, I'll gratefully listen to you.
>>>> 
>>>> The main question isn't "WHAT" but "HOW" - that's the discussion topic from
my point of view.
>>>> 
>>>> Regards,
>>>> Igor
>>>> 
>>>>> 27 сент. 2019 г., в 16:37, Nikolay Izhikov <nizhikov@apache.org>
написал(а):
>>>>> 
>>>>> Roman.
>>>>> 
>>>>>> Nikolay, Maxim, I understand that our arguments may not be as obvious

>>>>>> for you as it obvious for SQL team. So, please arrange your questions
in 
>>>>>> a more constructive way.
>>>>> 
>>>>> What is SQL team?
>>>>> I only know Ignite community :)
>>>>> 
>>>>> Please, share you knowledge in IEP.
>>>>> I want to join to the process of engine *selection*.
>>>>> It should start with the requirements to such engine.
>>>>> Can you write it in IEP, please?
>>>>> 
>>>>> My point is very simple:
>>>>> 
>>>>> 1. We made the wrong decision with H2
>>>>> 2. We should make a well-thought decision about the new engine.
>>>>> 
>>>>>> How many tickets would satisfy you?
>>>>> 
>>>>> You write about "issueS" with the H2.
>>>>> All I see is one open ticket.
>>>>> IEP doesn't provide enough information.
>>>>> So it's not about the number of tickets, it's about
>>>>> 
>>>>>> These two points (single map-reduce execution and inflexible optimizer)

>>>>>> are the main problems with the current engine.
>>>>> 
>>>>> We may come to the point when Calcite(or any other engine) brings us
third and other "main problems".
>>>>> This is how it happens with H2.
>>>>> 
>>>>> Let's start from what we want to get with the engine and move forward
from this base.
>>>>> What do you think?
>>>>> 
>>>>> 
>>>>> 
>>>>> В Пт, 27/09/2019 в 16:15 +0300, Roman Kondakov пишет:
>>>>>> Maxim, Nikolay,
>>>>>> 
>>>>>> I've listed two issues which show the ideological flaws of the current

>>>>>> engine.
>>>>>> 
>>>>>> 1. IGNITE-11448 - Open. This ticket describes the impossibility of

>>>>>> executing queries which can not be fit in the hardcoded one pass

>>>>>> map-reduce paradigm.
>>>>>> 
>>>>>> 2. IGNITE-6085 - Closed (won't fix) - This ticket describes the second

>>>>>> major problem with the current engine: H2 query optimizer is very

>>>>>> primitive and can not perform many useful optimizations.
>>>>>> 
>>>>>> These two points (single map-reduce execution and inflexible optimizer)

>>>>>> are the main problems with the current engine. It means that our
engine 
>>>>>> is currently  suitable for execution only a very limited subset of
the 
>>>>>> typical SQL queries. For example it can not even run most of the
TPC-H 
>>>>>> benchmark queries because they don't fit to the simple map-reduce
paradigm.
>>>>>> 
>>>>>>> All I see is links to two tickets:
>>>>>> 
>>>>>> How many tickets would satisfy you? I named two. And it looks like
it is 
>>>>>> not enough from your point of view. Ok, so how many is enough? The
set 
>>>>>> of problems caused by listed above tickets is infinite, therefore
I can 
>>>>>> not create a ticket for each of them.
>>>>>>> Tech details also should be added.
>>>>>> 
>>>>>> Tech details are in the tickets.
>>>>>> 
>>>>>>> We can't discuss such a huge change as an execution engine replacement
with descrition like:
>>>>>>> "No data co-location control, i.e. arbitrary data can be returned
silently" or
>>>>>>> "Low control on how query executes internally, as a result we
have limited possibility to implement improvements/fixes."
>>>>>> 
>>>>>> Why not? Don't you understand these problems? Or you don't think
this is 
>>>>>> a problem?
>>>>>> 
>>>>>>> Let's make these descriptions more specific.
>>>>>> 
>>>>>> What do you mean by "more specific"? What is the criteria of the

>>>>>> specific description?
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Nikolay, Maxim, I understand that our arguments may not be as obvious

>>>>>> for you as it obvious for SQL team. So, please arrange your questions
in 
>>>>>> a more constructive way.
>>>>>> 
>>>>>> Thank you!
>>>> 
>>>> 
>> 
>> 


Mime
View raw message