calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <>
Subject Re: [Avatica] Separating wire message from service implementation
Date Sat, 08 Aug 2015 00:40:10 GMT
Hrmph. Well, fun fact: that JSON serialization for pb3 apparently isn't 
implemented yet..

Josh Elser wrote:
> :) I hear you. I don't think I have anything putting me in one camp or
> the other aside from potential worries.
> I can respect the Thrift "doing too much" notion -- kind of half of my
> concern in using something like Thrift just to get object serialization
> stuff (now I'm thinking I should help separate that out within thrift!)
> I've used Thrift since the 0.6 timeframe and I don't remember once
> coming across an issue where object serialization wasn't handled correctly.
> I'll have to poke around pb3 docs some more to see if I can find some
> talk on compatibility. I think at this point I'm leaning towards
> protobuf, but we'll see.
> Thanks again for taking the time to write up your thoughts, Jesse!
> Jesse Yates wrote:
>> Yup, totally get it. Honestly, I have no idea what the compat story is
>> supposed to be for proto3. At the same time, I've got no hands on
>> experience w/ Thrift either, beyond dabbling a little bit years ago, so I
>> don't know what their on-the-wire-compat story is either :) I'm not as
>> huge
>> of a fan of thrift in general though because it tries to do too much IMO
>> If we do go the thrift route, but want to support grpc, we could do the
>> work to make a thrift adapter, so picking one or the other now doesn't
>> really preclude doing cool stuff later.
>> In short, it doesn't really matter? At least not to me.
>> On Thu, Aug 6, 2015 at 3:57 PM Josh Elser<> wrote:
>>> Thanks for the feedback, Jesse!
>>> To clarify, the compatibility I'm most worried about is backwards compat
>>> in future versions of protobuf3. I hadn't considered v2 support into the
>>> equation -- figured v3 is the starting point like you say. I know the
>>> protobuf 2.4 to 2.5 upgrade in Hadoop/HBase land was rather scary that
>>> required a lot of lock-step movement across many big projects. This is
>>> my biggest fear with protobuf.
>>> Despite fear of derailing conversation, grpc definitely has some nice
>>> features built in (and implementing a service is definitely slick). I
>>> think it's definitely a "later" since the single endpoint presently used
>>> doesn't really require a complex service definition (with evolving args,
>>> etc). My gut reaction is to wait to adopt such software until there is a
>>> definite understanding of what the service would look like (which I
>>> assume will become clearer as development continues).
>>> Jesse Yates wrote:
>>>> +1 for proto3. mostly for the later move to using grpc, which lets us
>>>> leverage a battle tested framework for doing real protobuf services
>>>> with
>>>> fast streaming (how nice!).
>>>> proto3 can be backwards compat in some ways (eg. map implementation
>>>> done
>>>> manually in proto2). I don't think its a huge issue since there are
>>>> currently no backwards compat guarantees anyways AFAIK.
>>>> On Thu, Aug 6, 2015 at 3:11 PM Josh Elser<> wrote:
>>>>> Hi,
>>>>> I have some ideas which I'd like to try to working on to improve
>>>>> Avatica. My direct interest is in support of the Phoenix Query Server
>>>>> which builds on top of Avatica.
>>>>> One of the big areas of improvement that stands out to me is the
>>>>> Jackson-annotated Response/Request classes. First, we'd want to make
>>>>> sure these classes are stable (in terms of fields/attributes) so that,
>>>>> as each object evolves, we prevent existing clients from breaking.
>>>>> Second, it's desirable that we have some representation of these
>>>>> objects
>>>>> in a manner that's accessible to non-Java clients.
>>>>> Ideally, preserving JSON as the serialized representation of these
>>>>> classes is also desirable as it's always pleasant to actually be
>>>>> able to
>>>>> read the data going over the wire when necessary.
>>>>> Given these goals, two tools stood out to me as candidates that would
>>>>> solve this problem well without extra effort in the transport: Thrift
>>>>> and Protobuf v3.
>>>>> Using Thrift only for object generation is a bit overkill, but this is
>>>>> an area of Thrift that has been rock-solid in the years that I've used
>>>>> it. Classpath issues with Thrift are always a concern as multiple
>>>>> versions of Thrift tend to break terribly (at least when performing
>>>>> Thrift RPCs -- I wouldn't necessarily trust it to be safe for only
>>>>> object serialization). It's easy to serialize Thrift objects to
>>>>> JSON and
>>>>> we can generate these classes for other languages.
>>>>> I would call Protobuf v3 very young, although it does appear to fit
>>>>> the
>>>>> bill. Protobuf v2 has had a track record of being a pain when
>>>>> different
>>>>> versions of the library exist. For example, the Java classes generated
>>>>> by protobuf 2.x would fail to work with protobuf 2.y. I assume there's
>>>>> not enough evidence on protobuf v3 to say whether this will
>>>>> continue to
>>>>> be a problem. JSON-representation of objects and cross-language
>>>>> support
>>>>> are both natively supported (best as I can tell). Hypothetically,
>>>>> adopting protobufv3 could also have some extra benefits down the road
>>>>> such as leveraging grpc to replace Jetty, but that is an entirely
>>>>> different discussion for later :)
>>>>> If you've gotten this far, I'd humbly ask for any opinions on the
>>>>> technology choices. Any opinions for or against one or the other? Any
>>>>> similar experiences?
>>>>> Thanks!
>>>>> - Josh

View raw message