calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julian Hyde <jh...@apache.org>
Subject Re: Materialized view rewriting
Date Fri, 26 Feb 2016 18:13:30 GMT
Rajat,

I’ll second what Maryann says. Please chime in on https://issues.apache.org/jira/browse/CALCITE-1101
<https://issues.apache.org/jira/browse/CALCITE-1101>. If it doesn’t make things easier
for Quark we probably shouldn’t be doing it.

Julian

> On Feb 26, 2016, at 8:39 AM, Maryann Xue <maryann.xue@gmail.com> wrote:
> 
> Thank you for pointing out another way of defining materializations, Rajat!
> 
> We had some discussion again about this interface, and Julian opened a JIRA
> https://issues.apache.org/jira/browse/CALCITE-1101.
> 
> The main problems are:
> 
> 1. The life cycle of the materializations. In Phoenix (we don't know about
> other projects, so welcome more comments), materializations are used to
> model secondary indices, which are in fact another type of Phoenix tables.
> So materializations for Phoenix should have exactly the same life cycle as
> its enclosed PhoenixSchema, which is a snapshot of all current table
> definitions as of the timestamp for a specific JDBC statement.
> 
> 2. Right now for calling defineMaterialization method, we need to take a
> whole lot of trouble to get the CalciteSchema object which in fact should
> be internal to Calcite code.
> 
> 3. The right point of time for defining/creating those materialized views.
> Whether for Quark or Phoenix, we need to make sure that we call
> defineMaterialization at the exact right point of time, which is after the
> schema is loaded and before the planner tries to collect and use them.
> Again this had better be something taken care of by Calcite instead of
> carefully maintained by the users.
> 
> 
> Let's follow up on that JIRA though.
> 
> 
> 
> Thanks,
> Maryann
> 
> On Fri, Feb 26, 2016 at 5:50 AM, Rajat Venkatesh <rvenkatesh@qubole.com>
> wrote:
> 
>> In Quark, dont use hooks to define materializations. We use a tablefactory
>> [1] to defer until the schema is loaded[2].
>> 
>> 1.
>> 
>> https://github.com/qubole/quark/blob/master/optimizer/src/main/java/com/qubole/quark/planner/MetadataSchema.java#L85
>> 2.
>> 
>> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/materialize/MaterializationService.java#L132
>> 
>> Hopefully, I've understood your problem correctly.
>> 
>> 
>> On Thu, Feb 25, 2016 at 10:16 PM Maryann Xue <maryann.xue@gmail.com>
>> wrote:
>> 
>>> Hi Michael,
>>> 
>>> We had a little difficulty defining our secondary indices as materialized
>>> views with our Schema SPI implementation too, and this made the code
>> pretty
>>> hacky. In order to call that defineMaterialization method, we hold the
>>> parent SchemaPlus object within our own Schema impl object so that we can
>>> get its own corresponding SchemaPlus object by calling "parentSchema
>>> .getSubSchema(this.name)" later on. We do this after the schema/table
>>> resolving phase (that is when the entire schema tree incl. your own
>> schema
>>> objects have been initiated) and call defineMaterialization for each
>>> secondary index under each subSchema. We add a hook in "Hook.TRIMMED" for
>>> this, which sounds pretty weird, but this is exactly a point after you
>> have
>>> the whole schema tree ready and before the materializations are asked for
>>> by the planner.
>>> 
>>> Anyway, I do hope the interface can be modified to avoid all this
>> trouble.
>>> 
>>> 
>>> Thanks,
>>> Maryann
>>> 
>>> On Thu, Feb 25, 2016 at 9:24 AM, Michael Mior <mmior@uwaterloo.ca>
>> wrote:
>>> 
>>>> Any suggestions on the best place to hook in and add the materialized
>>>> views? It seems like doing so requires the SchemaPlus object
>>> corresponding
>>>> to the current schema. The current best approach I see is to save a
>>>> reference to the parent schema and then pull out the appropriate
>>> SchemaPlus
>>>> object in getTableMap. This seems like a bit of a hack though.
>>>> 
>>>> --
>>>> Michael Mior
>>>> mmior@uwaterloo.ca
>>>> 
>>>> 2016-02-24 17:22 GMT-05:00 Julian Hyde <jhyde@apache.org>:
>>>> 
>>>>> By the way, interesting that you are interested in Cassandra and
>>>>> materialized views. Cassandra announced materialized view support
>>>>> recently[1] but they solved only half of a problem (not an
>>>>> insignificant half, I hasten to add), namely materialized view
>>>>> maintenance. They don't transparently substitute them into the query
>> -
>>>>> you have to reference the materialized view explicitly in y our query
>>>>> - so in my book they've not delivered materialized view support. If
>>>>> you're planning to deliver REAL materialized view support to
>> Cassandra
>>>>> that would be awesome.
>>>>> 
>>>>> Julian
>>>>> 
>>>>> [1]
>>>>> 
>>> http://www.datastax.com/dev/blog/new-in-cassandra-3-0-materialized-views
>>>>> 
>>>>> 
>>>>> On Wed, Feb 24, 2016 at 2:17 PM, Julian Hyde <jhyde@apache.org>
>> wrote:
>>>>>> As is typical for complex pieces of code like this, the
>> documentation
>>>>>> is in the code (and the unit test). It's probably not what you
>> wanted
>>>>>> to hear, but the code mutates quite fast and so if we'd written a
>>>>>> design doc a few months ago it would be partially inaccurate.
>>>>>> 
>>>>>> I, Maryann Xue and Amogh Margoor are the main authors of this code.
>>>>>> 
>>>>>> Suggest you find a relevant test case in MaterializationTest (or
>>> write
>>>>>> a new one) and run it with trace enabled and/or in a debugger. You
>>>>>> will see the process of matching an expression to a MV bottom up
if
>>>>>> you watch each call to UnifyRule.unify.
>>>>>> 
>>>>>> Julian
>>>>>> 
>>>>>> 
>>>>>> On Wed, Feb 24, 2016 at 1:40 PM, Michael Mior <mmior@uwaterloo.ca>
>>>>> wrote:
>>>>>>> Is there any documentation anywhere on how the current
>>> implementation
>>>> of
>>>>>>> query rewriting for materialized views work? Mostly I'm referring
>>>>>>> to MaterializedViewSubstitutionVisitor. There's a lot of code
to
>>>> digest
>>>>>>> with not a lot of documentation and it would be helpful to have
a
>>>>> reference
>>>>>>> to refer. Thanks!
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> --
>>>>>>> Michael Mior
>>>>>>> mmior@uwaterloo.ca
>>>>> 
>>>> 
>>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message