gora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renato MarroquĂ­n Mogrovejo <renatoj.marroq...@gmail.com>
Subject Re: [PROPOSAL] GORA-174
Date Sat, 16 Feb 2013 18:39:50 GMT
Hi all,

2013/2/15 Lewis John Mcgibbney <lewis.mcgibbney@gmail.com>:
> Hi,
> OK so in an attempt to try and close this off I am proposing the following.
> We follow up on Alfonso's road map... which is as follows
> *Proposals*
>    1. Create a configuration opt-in option for a DataStore to write
>    null-plus-onetype-unions without the index byte and delete the column when
>    null. Process properly when reading. This allows other systems to directly
>    read data easily.

Ok, so I think I am understanding now. We are just trying to solve the
null-plus-onetype-unions? Ok then, Alfonso's proposal is the way to
go. Ignore the field when it is null, and persist it when it is not
null. But my point was on supporting various-type unions. I know that
HBase will be able to support any type of unions because it serializes
data using Avro, but we have to decide on how other non-avro-based
data stores will handle any type of unions.

>    2. Create a *deprecated* configuration option that will parse schemas
>    ignoring unions and nulls, and will work like Gora 0.2.1. This will enable
>    anyone using pre 0.3 versions of Gora to upgrade.

I think 0.2.1 with Alfonso's patch does handle this
null-plus-onetype-unions correctly. So +1 on this one.

>    3. Take on board Renato's suggestions regarding the CassandraStore.
>    @Renato, if possible it would be great for you to comment on GORA-174 with
>    your precise objectives.

I think I have been confusing things here. GORA-174 is for supporting
null-plus-onetype-unions, and I was addressing support for
various-type-unions for non-avro-based data stores which is a
different issue.

> *Concerns*
>    1. AccumuloStore option as we do not seem to have a proposal to adapt
>    the AccumuloStore logic yet.

The changes made into the compiler will alleviate further changes in
other data stores. We should open another issue in JIRA to clarify
that we need to address this in a near future.

>    2. AFAIK we DO NOT need to worry about DynamoDB as we do not use Avro at
>    all but if you could confirm Renato that would be great.

This is one thing I have been meaning to get around to. DynamoDB is
not avro-based but Gora heavily relies in avro structures to maintain
in memory data structures to then flush them into specific back-ends
in a batch manner. As Amazon didn't provide any cost benefit for
writing directly or in batch operations for DynamoDB, the
Gora-DynamoDB does not support batch operations (operations performed
when we do a "flush"). At the end of last year Amazon started
providing support for batch operations for DynamoDB which now makes
sense to build all the other features within the Gora-DynamoDB module
... *sight* ... that was a long explanation (: I will address this in
separate issues.

>    3. AvroStore... what is going on here? Do we need to make changes
>    locally to the store or are these already supported? My feeling is that
>    they are not... unless of course Alfonso's existing patches actually
>    address this. Can you please confrim Alfonso?

IMHO we should probably make a roadmap for the AvroStore so we can
take this store as a model for other data stores (as most of them rely
on it), and get that Avro patch in!!! Maybe we should join strengths
and tackle that issue after closing this one.

> *Actions*
>    1. Address Proposals[1,2,3]: Produce concrete patches and get them
>    submitted to GORA-174


>    2. Re-base on Concerns[1,2,3] and of course add to them with input from
>    you guys.


>    3. Test the functionality and comment on the issues opened by Renato
>    recently e.g. GORA-206 & 207.

+1 I will do, and open different JIRA issues for supporting
various-type-unions inside other data stores, and then we can discuss
each case separately.

>    4. Push the 0.3 RC.


> *Notes*
> It should be noted that we have seen little or no non-backwards compatible
> changes in the Gora API since the project was accepted into the Apache
> Incubator. Although we should aim to always maintain backwards
> compatibility, we should not shy away from engaging with proposals to move
> the code base forward.

+1 with this one as well. But again I think we should differ between
backward compatibility and new features, as the null-one-type-union
case. I know 0.2.1 release will only ignore them, or log a message
saying that union data type is not FULLY supported in this version
which is IMHO a  new feature.

> The PMC and Committer base as well as users determine in which direction
> Gora goes. Without implementing some of the ideas Gora will go no where.
> On that note, have a great weekend everyone, I am looking forward to
> ironing out GORA-174 as it has dominated the Gora agenda somewhat recently.


Renato M.

> Thanks
> Lewis
> --
> *Lewis*

View raw message