thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jens Geyer" <jensge...@hotmail.com>
Subject Re: updating to longer ints
Date Fri, 15 Mar 2013 08:07:56 GMT
Hi,

I like the idea of the data types getting promoted to bigger ones on read, 
but I'm still afraid it won't work. Lets assume a service like this:

    struct foo {
      1: i16 bignumber,
      2: i16 anotherone,
      3: i16 third
    }

    service bar {
      foo Something( 1: foo input)
    }

Now, after some time, you find out, that i16 is way too small, so you change 
foo{} to hold all i32's.

First, the real issue is, when not all clients and servers are under your 
control and can be updated at once, it will break. The generated code holds 
declarations for all variables, so it can't be promoted on the fly only as 
far as the current data structures allow this.

But OTOH the idea has some good aspects, since it brings in the idea that 
the size of the variable must not necessarily be equal to the amount of the 
bytes on the wire, e.g. a int32 = 0x00001234 could be read/written as i16 
without losing information. The only thing that may be affected here is the 
performance, because the code has to check for appropriate write and read 
sizes.

Jens


-----Urspr√ľngliche Nachricht----- 
From: Will Pierce
Sent: Thursday, March 14, 2013 10:47 PM
To: user@thrift.apache.org
Subject: Re: updating to longer ints

Definitely not an easy problem.  On the wire these will both encode as the
same field ID (1) and a type ID of 6 for the i16 or a type ID of 8 for the
i32.  Most (all?) thrift language bindings check to ensure the type ID for
a field matches the expected type ID for that particular field, silently
skipping past any field with mismatched type IDs.

George Chung's suggestion is the safest/cleanest approach.

I had exactly this same problem where I needed to expand from i32 to i64.
It was a "flag day" cutover for me, since I had control of all the clients
and servers.  But I can see how you frequently can't upgrade all the client
code out in the wild.  (Or data files, if these are persisted.)

Rather than try to solve the broader problem of versioning, maybe we should
consider adding optional behavior to the decoder to permit
*type-promotion*when the field's on-wire type can be safely decoded
into the expected type?

Provided that the expected type can faithfully represent all the values of
the wire type, it seems like a way to follow the "TCP" pattern of being
permissive in what you receive, but strict in what you send.

In terms of thrift TTypes, we could allow a decoder to promote:
bool > i08 > i16 > i32 > (double or i64)

This wouldn't solve every problem - like sending a response struct with an
i32 field to a client than expects i16.  No go there.  This would be only
useful in cases of having to read client input from a mixed bag of old and
new clients, where the server response struct doesn't need any wider
fields.  I'm not sure if that is your use-case...

Those using thrift for serialization to/from disk, I would imagine
type-promotion could save a lot of effort in re-encoding any old files that
use narrower numeric types than newer files...

Thoughts? 


Mime
View raw message