thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: updating to longer ints
Date Fri, 15 Mar 2013 15:05:08 GMT
On Fri, Mar 15, 2013 at 4:07 AM, Jens Geyer <jensgeyer@hotmail.com> wrote:
> Hi,
>
> I like the idea of the data types getting promoted to bigger ones on read,
> but I'm still afraid it won't work. Lets assume a service like this:
>
>    struct foo {
>      1: i16 bignumber,
>      2: i16 anotherone,
>      3: i16 third
>    }
>
>    service bar {
>      foo Something( 1: foo input)
>    }
>
> Now, after some time, you find out, that i16 is way too small, so you change
> foo{} to hold all i32's.
>
> First, the real issue is, when not all clients and servers are under your
> control and can be updated at once, it will break. The generated code holds
> declarations for all variables, so it can't be promoted on the fly only as
> far as the current data structures allow this.
>
> But OTOH the idea has some good aspects, since it brings in the idea that
> the size of the variable must not necessarily be equal to the amount of the
> bytes on the wire, e.g. a int32 = 0x00001234 could be read/written as i16
> without losing information. The only thing that may be affected here is the
> performance, because the code has to check for appropriate write and read
> sizes.

I think TCompactProtocol uses var length ints, achieving something
similar to what you mentioned.  Not that this would help w/ the
original issue.

>
> Jens
>
>
> -----Urspr√ľngliche Nachricht----- From: Will Pierce
> Sent: Thursday, March 14, 2013 10:47 PM
> To: user@thrift.apache.org
> Subject: Re: updating to longer ints
>
>
> Definitely not an easy problem.  On the wire these will both encode as the
> same field ID (1) and a type ID of 6 for the i16 or a type ID of 8 for the
> i32.  Most (all?) thrift language bindings check to ensure the type ID for
> a field matches the expected type ID for that particular field, silently
> skipping past any field with mismatched type IDs.
>
> George Chung's suggestion is the safest/cleanest approach.
>
> I had exactly this same problem where I needed to expand from i32 to i64.
> It was a "flag day" cutover for me, since I had control of all the clients
> and servers.  But I can see how you frequently can't upgrade all the client
> code out in the wild.  (Or data files, if these are persisted.)
>
> Rather than try to solve the broader problem of versioning, maybe we should
> consider adding optional behavior to the decoder to permit
> *type-promotion*when the field's on-wire type can be safely decoded
> into the expected type?
>
> Provided that the expected type can faithfully represent all the values of
> the wire type, it seems like a way to follow the "TCP" pattern of being
> permissive in what you receive, but strict in what you send.
>
> In terms of thrift TTypes, we could allow a decoder to promote:
> bool > i08 > i16 > i32 > (double or i64)
>
> This wouldn't solve every problem - like sending a response struct with an
> i32 field to a client than expects i16.  No go there.  This would be only
> useful in cases of having to read client input from a mixed bag of old and
> new clients, where the server response struct doesn't need any wider
> fields.  I'm not sure if that is your use-case...
>
> Those using thrift for serialization to/from disk, I would imagine
> type-promotion could save a lot of effort in re-encoding any old files that
> use narrower numeric types than newer files...
>
> Thoughts?

Mime
View raw message