gora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kazuomi Kashii <kazu...@kashii.net>
Subject [DISCUSS] gora-cassandra serialization spec
Date Fri, 06 Jul 2012 18:40:16 GMT
I wrote GORA-142-v3.patch that supports several new types of
serialization for gora-cassandra.
https://issues.apache.org/jira/browse/GORA-142

Since I am not familiar with other implementation such as gora-hbase,
I'd like to hear your opinions on serialization spec, especially for
variable length array.

Gora uses Avro for schema definition,
but I noticed that gora-cassandra uses its own serialization based Hector.
For instance, serialization of integer is totally different between Avro
(zig-zag) and gora-cassandra (Hector).
Considering Cassandra's pre-defined validation classes and comparators,
I think Hector's serialization is better than Avro's one at gora-cassandra,
so, my implementation of GORA-142 patch is based on Hector's serializers.

For ARRAY support, I implemented GORA-138 patch first with Super CF in
the same way as RECORD or MAP.
https://issues.apache.org/jira/browse/GORA-138
As Enis mentioned at GORA-138, we may want another implementation with
single column for reasonably short arrays,
so GORA-142 patch supports ARRAY with single column implementation.

For fixed length array, single column can store multiple elements just
adding them sequentially.
However, for variable length array such as STRING or BYTES,
it is impossible to retrieve each value if just values are stored
sequentially,
so GORA-142 patch implementation contains the size of element as INTEGER
before each actual value.
For instance, ["ABCDE", "abc", "1234"] is stored as
00 00 00 05 41 42 43 44 45 00 00 00 03 61 62 63 00 00 00 04 31 32 33 34

If there is no obligation, I will commit GORA-142 patch with above
serialization spec later once it is ready.

Regards,
-Kaz


On 7/6/12 11:05 AM, Kazuomi Kashii wrote:
> +1 for new release
>
> 1) I committed GoraCompiler.java for GORA-143-v2.patch last night.
>   Since this is my first svn commit, I think it should be reviewed.
>
> 2) I have not committed GORA-142 patch for gora-cassandra yet.
>   Before that, I'd like to ask the team about serialization spec, so I
> will send another e-mail on this matter.
>   I am OK for new release without GORA-142 patch.
>   Depending on the discussion, I will commit GORA-142 before or after
> new release.
>
> -Kaz
>
>
> On 7/6/12 10:29 AM, Henry Saputra wrote:
>> +1 for new release, there are a lot fixes for Cassandra support so
>> should be good time for new release.
>>
>> 0.3 or 0.2.1?
>>
>> - Henry
>>
>> On Thu, Jul 5, 2012 at 11:06 PM, Mattmann, Chris A (388J)
>> <chris.a.mattmann@jpl.nasa.gov> wrote:
>>> +1 to roll release and I'll also throw my name into the hat to release it.
>>>
>>> Let me know.
>>>
>>> Thanks!
>>>
>>> Cheers,
>>> Chris
>>>
>>> On Jul 5, 2012, at 12:30 PM, Lewis John Mcgibbney wrote:
>>>
>>>> Hi,
>>>>
>>>> As the GSoC project is moving along nicely and it's been some 3 or so
>>>> months since the 0.2 release I was thinking about drumming up support
>>>> for another (possibly even 0.2.1) release?
>>>>
>>>> We have some 15 issues which have been addressed in the development
>>>> drive since 0.2 was released and I for one have not had quite as much
>>>> time as I would have liked recently to put serious time into Gora.
>>>>
>>>> What do you guys think? I am more than happy to work as RM again if required.
>>>>
>>>> Thank you in advance
>>>>
>>>> Lewis
>>>>
>>>> --
>>>> Lewis
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW:   http://sunset.usc.edu/~mattmann/
>>> Phone: +1 (818) 354-8810
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>



Mime
View raw message