calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julian Hyde <>
Subject Re: [Avatica] Separating wire message from service implementation
Date Mon, 10 Aug 2015 22:01:01 GMT
If my understand is correct, this decision is an easy one to make, because whatever we choose,
we still comply with the specification.

Avatica specifies its API at two levels: (1) the Java API (Meta), (2) the JSON requests/responses
going over the wire.

I would accept any transport that complies with both of those levels. In principle we could
write a JSON-over-HTTP-marshalled-via-Jackson transport, specifying the mappings elsewhere
than in the Meta classes, and someone else could use another transport.

Is it feasible to achieve this? 


> On Aug 7, 2015, at 5:40 PM, Josh Elser <> wrote:
> Hrmph. Well, fun fact: that JSON serialization for pb3 apparently isn't implemented yet..
> Josh Elser wrote:
>> :) I hear you. I don't think I have anything putting me in one camp or
>> the other aside from potential worries.
>> I can respect the Thrift "doing too much" notion -- kind of half of my
>> concern in using something like Thrift just to get object serialization
>> stuff (now I'm thinking I should help separate that out within thrift!)
>> I've used Thrift since the 0.6 timeframe and I don't remember once
>> coming across an issue where object serialization wasn't handled correctly.
>> I'll have to poke around pb3 docs some more to see if I can find some
>> talk on compatibility. I think at this point I'm leaning towards
>> protobuf, but we'll see.
>> Thanks again for taking the time to write up your thoughts, Jesse!
>> Jesse Yates wrote:
>>> Yup, totally get it. Honestly, I have no idea what the compat story is
>>> supposed to be for proto3. At the same time, I've got no hands on
>>> experience w/ Thrift either, beyond dabbling a little bit years ago, so I
>>> don't know what their on-the-wire-compat story is either :) I'm not as
>>> huge
>>> of a fan of thrift in general though because it tries to do too much IMO
>>> If we do go the thrift route, but want to support grpc, we could do the
>>> work to make a thrift adapter, so picking one or the other now doesn't
>>> really preclude doing cool stuff later.
>>> In short, it doesn't really matter? At least not to me.
>>> On Thu, Aug 6, 2015 at 3:57 PM Josh Elser<> wrote:
>>>> Thanks for the feedback, Jesse!
>>>> To clarify, the compatibility I'm most worried about is backwards compat
>>>> in future versions of protobuf3. I hadn't considered v2 support into the
>>>> equation -- figured v3 is the starting point like you say. I know the
>>>> protobuf 2.4 to 2.5 upgrade in Hadoop/HBase land was rather scary that
>>>> required a lot of lock-step movement across many big projects. This is
>>>> my biggest fear with protobuf.
>>>> Despite fear of derailing conversation, grpc definitely has some nice
>>>> features built in (and implementing a service is definitely slick). I
>>>> think it's definitely a "later" since the single endpoint presently used
>>>> doesn't really require a complex service definition (with evolving args,
>>>> etc). My gut reaction is to wait to adopt such software until there is a
>>>> definite understanding of what the service would look like (which I
>>>> assume will become clearer as development continues).
>>>> Jesse Yates wrote:
>>>>> +1 for proto3. mostly for the later move to using grpc, which lets us
>>>>> leverage a battle tested framework for doing real protobuf services
>>>>> with
>>>>> fast streaming (how nice!).
>>>>> proto3 can be backwards compat in some ways (eg. map implementation
>>>>> done
>>>>> manually in proto2). I don't think its a huge issue since there are
>>>>> currently no backwards compat guarantees anyways AFAIK.
>>>>> On Thu, Aug 6, 2015 at 3:11 PM Josh Elser<>
>>>>>> Hi,
>>>>>> I have some ideas which I'd like to try to working on to improve
>>>>>> Avatica. My direct interest is in support of the Phoenix Query Server
>>>>>> which builds on top of Avatica.
>>>>>> One of the big areas of improvement that stands out to me is the
>>>>>> Jackson-annotated Response/Request classes. First, we'd want to make
>>>>>> sure these classes are stable (in terms of fields/attributes) so
>>>>>> as each object evolves, we prevent existing clients from breaking.
>>>>>> Second, it's desirable that we have some representation of these
>>>>>> objects
>>>>>> in a manner that's accessible to non-Java clients.
>>>>>> Ideally, preserving JSON as the serialized representation of these
>>>>>> classes is also desirable as it's always pleasant to actually be
>>>>>> able to
>>>>>> read the data going over the wire when necessary.
>>>>>> Given these goals, two tools stood out to me as candidates that would
>>>>>> solve this problem well without extra effort in the transport: Thrift
>>>>>> and Protobuf v3.
>>>>>> Using Thrift only for object generation is a bit overkill, but this
>>>>>> an area of Thrift that has been rock-solid in the years that I've
>>>>>> it. Classpath issues with Thrift are always a concern as multiple
>>>>>> versions of Thrift tend to break terribly (at least when performing
>>>>>> Thrift RPCs -- I wouldn't necessarily trust it to be safe for only
>>>>>> object serialization). It's easy to serialize Thrift objects to
>>>>>> JSON and
>>>>>> we can generate these classes for other languages.
>>>>>> I would call Protobuf v3 very young, although it does appear to fit
>>>>>> the
>>>>>> bill. Protobuf v2 has had a track record of being a pain when
>>>>>> different
>>>>>> versions of the library exist. For example, the Java classes generated
>>>>>> by protobuf 2.x would fail to work with protobuf 2.y. I assume there's
>>>>>> not enough evidence on protobuf v3 to say whether this will
>>>>>> continue to
>>>>>> be a problem. JSON-representation of objects and cross-language
>>>>>> support
>>>>>> are both natively supported (best as I can tell). Hypothetically,
>>>>>> adopting protobufv3 could also have some extra benefits down the
>>>>>> such as leveraging grpc to replace Jetty, but that is an entirely
>>>>>> different discussion for later :)
>>>>>> If you've gotten this far, I'd humbly ask for any opinions on the
>>>>>> technology choices. Any opinions for or against one or the other?
>>>>>> similar experiences?
>>>>>> Thanks!
>>>>>> - Josh

View raw message