asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ildar Absalyamov <>
Subject Re: Metadata changes
Date Tue, 15 Dec 2015 00:22:23 GMT
It is true that we don’t want the information about “schema” of metadata record to be
scattered all over, but there is a single place where this information is always correct are
describes the current format of the records - tupleTranslators, which serialize\deserializes
metadata records into/from objects. Right now they look like bunch of boilerplate code, however
if they were to be refactored with handy util methods that would have been the best “documentation”
for the current state of the world, no?

> On Dec 14, 2015, at 13:19, Mike Carey <> wrote:
> I think we started out as a bunch of relation-heads, modeling things after the system
catalogs found in our favorite RDBMSs.  Their record types are open AFAIK - but I think Steven
was saying that it feels like this is a use case whose field(s) should be predeclared (and
using the term "closed field" to mean predeclared field).  I agree.  I think to the extent
that we can anticipate the need for a given bit of metadata, it's useful to have it be there
in the known schema; but, when we move into the era when we cannot make changes to the known
schema part of the catalogs, because we have dependent customers who'd be hurt by that, we
will start adding more fields that are not predeclared but only documented.  The schema for
the metadata is really there more as documentation than as being "truly necessary"...
> When we first started out we were just not thinking in an "open-minded" way on the catalog
side of things.....  :-)
> Cheers,
> Mike
> On 12/14/15 11:53 AM, Till Westmann wrote:
>> Let me prefix this with that statement that I haven’t looked into our metadata
storage implementation.
>> But looking at the relatively frequent changes to the metadata format it seems that
we should either
>> a) make the types for metadata records open or
>> b) be very careful in designing the next (and future) revision(s) of it.
>> I don’t know why the metadata storage was done the way it is now and what the considerations
were when we did that.
>> Does anybody remember those (or is there even a document that contains the design
and rationale)?
>> Thanks,
>> Till
>> On 14 Dec 2015, at 11:35, Steven Jacobs wrote:
>>> It could easily be done as reverse-compatible, but my thinking was that
>>> this is the "wrong" choice. I can easily make the datatype dataverse an
>>> open field for the metadata. The question is, why do we have open vs closed
>>> fields for metadata at all? If it is okay for them to be open, should we
>>> get rid of the schema entirely? if it's not okay, then shouldn't this field
>>> be closed? From a design standpoint it seems that if reverse compatibility
>>> were not an issue than the field should be closed.
>>> Steven
>>> On Mon, Dec 14, 2015 at 11:30 AM, Ildar Absalyamov <
>>>> wrote:
>>>> If fix for 2) will break the backwards-compatibility 1) might do that as
>>>> well and be submitted together.
>>>> Now 2) was a long overdue problem, I don’t think there is any reason even
>>>> to try make changes backwards-compatible, because it was broken in the
>>>> first place.
>>>>> On Dec 14, 2015, at 11:16, Steven Jacobs <> wrote:
>>>>> It's a new attribute, but it's a closed field, which means it isn't
>>>>> backwards compatible.
>>>>> Steven
>>>>> On Mon, Dec 14, 2015 at 11:13 AM, Ian Maxon <> wrote:
>>>>>> For 1), I guess the question is whether it would be a backwards
>>>>>> compatible change. Since it's just a new attribute (right?...), and
>>>>>> is also sort of a new feature rather than a fix for something that
>>>>>> critically broken, I would tend toward putting it on master. If it's
>>>>>> not backwards compatible though maybe it needs more careful
>>>>>> consideration.
>>>>>> -Ian
>>>>>> On Mon, Dec 14, 2015 at 10:12 AM, Steven Jacobs <>
>>>> wrote:
>>>>>>> Hi all,
>>>>>>> I'm implementing a change so that datasets can use datatypes
>>>>>> alternate
>>>>>>> data verses (previously the type and set had to be from the same
>>>>>>> dataverse). Unfortunately this means another change for Dataset
>>>> Metadata
>>>>>>> (which will now store the dataverse for its type).
>>>>>>> As such, I had a couple of questions:
>>>>>>> 1) Should this change be thrown into the release branch, as it
>>>> another
>>>>>>> Metadata change?
>>>>>>> 2) In implementing this change, I've been looking at the Metadata
>>>>>> secondary
>>>>>>> indexes. I had a discussion with Ildar, and it seems the thread
>>>>>> Metadata
>>>>>>> secondary indexes being "hacked" has been lost. Is this also
>>>>>> that
>>>>>>> should get into the release? Is there anyone currently looking
at it?
>>>>>>> Steven
>>>> Best regards,
>>>> Ildar

Best regards,

View raw message