thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Craig Peterson <cpeter...@ancestry.com>
Subject RE: Using thrift IDL as the schema for arbitrary JSON messages
Date Wed, 08 Oct 2014 22:54:04 GMT
I made that json generator in jira as a way to make the idl available to other platforms without
having to write a full parser. My intention was really to provide an alternate view of the
idl that is a bit easier to consume. I am not clear exactly how it would be relevant to using
json as a wire protocol. It is not the same as the facebook thrift version which appears to
be a bit simpler. My intention was really to dump the entire parsed data structure the compiler
uses internally and make it available via json for other tools to use. 

I have since found a different way to do what I was working on at the time and stopped trying
to push on getting that patch merged. It would still be nice to have, since I can think of
a number of tools I would like to make, but I don't really want to write a custom IDL parser
for my platform of choice. I am not sure how well it will work with various IDL features like
imported files or complex default values or things like that.

-----Original Message-----
From: Roger Meier [mailto:roger@bufferoverflow.ch] 
Sent: 08 October 2014 4:49
To: Peter Neumark; dev@thrift.apache.org
Cc: user@thrift.apache.org
Subject: Re: Using thrift IDL as the schema for arbitrary JSON messages

Hi Peter

Quoting Peter Neumark <neumark.peter@gmail.com>:

> Thanks, Roger!
>
> I'm also posting this to the dev list, because I think that may be 
> more appropriate than user@.
Yes, that's why I'm cross posting now.

> The json generator is awesome (any estimate on when it will be 
> merged?),
there are two versions and I like to have one...
I'm ready to commit it as soon as I get proper response on my questions on that issue. Json
Schema is what i would like to have for the future.

> but that's not really what I was working on.
> The goal here is to ready an aribrary JSON message as thrift data.
Yes, I know. However using a schema would simplify this and that's why I started to talk about
the json generator.
-roger

>
> Pete
>
> On Wed, Oct 8, 2014 at 11:47 AM, Roger Meier <roger@bufferoverflow.ch>
> wrote:
>
>> Hi Peter
>>
>> Yes, JSON is worth to add and I see an option by using json schema as 
>> mentioned here:
>> https://issues.apache.org/jira/browse/THRIFT-2360
>>
>> best!
>> -roger
>>
>>
>> Quoting Peter Neumark <neumark.peter@gmail.com>:
>>
>>  Hi all,
>>>
>>> At Prezi, we currently use Thrift for a lot of the internal 
>>> communication between services. A lot of engineers would like to 
>>> keep using HTTP + JSON as means of communicating between services, but with some
sort of schema.
>>>
>>> After evaluating several alternatives, the best idea we had so far 
>>> was to use the thrift IDL as the schema, and beef up the existing 
>>> TSimpleJSONProtocol code to read arbitrary JSON messages.
>>>
>>> What I'm most interested in is whether or not someone else is 
>>> already doing this?
>>>
>>> I have a very early stage python implementation of such a protocol here:
>>> https://github.com/neumark/thriftmodel/blob/master/thriftmodel/
>>> TFlexibleJSONProtocol.py
>>>
>>> We'll probably create an implementation for Java as well.
>>> The most difficult issue with reading JSON is that a lot of existing 
>>> JSON protocols allow for several different types in a field. My 
>>> solution was to have an "unboxed union". To illustrate this with an 
>>> example, if we accept the following messages:
>>>
>>> {"id": 1, "expires": 32324234234}
>>> {"id": 1, "expires": "2014.10.07 10:34"}
>>> {"id": 1, "expires": false}
>>>
>>> Then our Thrift definition would be
>>>
>>> // Note: this is an unboxed union
>>> union { // or struct in my implementation because python has no union yet
>>>   1: i64 unix_timestamp,
>>>   2: string date_string,
>>>   3: bool expiration_switch
>>> }
>>>
>>> struct msg {
>>>   1: i64 id,
>>>   2: expiry expires
>>> }
>>>
>>> The "expires" field must be read "unboxed" so to speak.
>>> What's the best option for encoding this in the thrift IDL?
>>> Has anyone considered adding annotations to the IDL for similar purposes?
>>>
>>> Thanks,
>>> Peter
>>>
>>
>>
>>


Mime
View raw message