thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Duxbury <>
Subject Re: heterogeneous collections
Date Tue, 04 May 2010 18:45:46 GMT
On Tue, May 4, 2010 at 11:27 AM, Mayan Moudgill <> wrote:

> Hmmm...
> 1. The debate is about sending ONLY field sizes OR sending type identifiers
> for lot of objects.
>  Consider the encoding of
> struct {1:i32 A, 2: list<i16> {B0,B1,B2}
> In TBinaryProtocol it gets encoded as:
>  T_STRUCT 1 T_I32 a_val 2 T_LIST 3 T_I16 b0_val b1_val b2_val T_STOP
> With a strongly typed system where the only thing allowed is different
> numbers of fields, the encoding would become
>  1 B=4 a_val 2 B=6 3 b0_val b1_val b2_val T_STOP
> where B= are the byte counds for the fields.

The idea of eliding type information from the serialized form is something
I've been thinking about for a while, and apparently something that David
explored in the "dense" protocol in the past. However, I think we can do a
lot better than the dense protocol if we go down the road of pluggable
serializers (

> 2. In thrift, we need the byte counts to skip unknown fields. So, we don't
> really need to know the actual byte count, but we need to know when a field
> will end. Note also that during the demarshalling process, at any one time,
> there will be at most one field being skipped. Also, fields can be managed
> as a stack.
> Assume one wanted to assemble data in chunks of 64KB. In that case, when a
> block 64KB had been assembled, some fields might still be being written. Of
> these fields, some would get fully written ("completed") in the next 64KB,
> and some would still be pending. We will keep information about the fields
> completed in a 64KB block in the beginning of the block. The completed-field
> information field will look like:
> * Depth of stack at beginning of block
> * number of pops
> * list of byte-offset
> So:
> - writeFieldBegin() pushes field onto a stack, and reserves 2B for the
> size; intially set value to 65535
> - writeFieldEnd() pops field from a stack; if the entire field is contained
> in a 64KB block, write byte offset  to field. If the field is from previous
> block, append byte offset to completed-information field.
> When skipping, either the skip is > 65535, in which case skip to offset
> within block, or it is 65535, in which case skip to the next block, read the
> completed-field info, see if the skipped field would get popped off (i.e.
> completed). If yes, skip to the offset recorded in the byte-offset list,
> else skip to end of block and repeat.

I don't mean to discourage you from thinking about more optimal structures,
but man, this doesn't strike you as overcomplicated? I can tell you from
experience that managing a stack of metadata (in the compact protocol) is
non-trivial both in terms of complexity and performance - and that's just
field IDs.

Additionally, including the type information gives us more than just the
ability to skip correctly. It also makes the serialized data fully
described. This makes it easy to debug it when you might think something is
going wrong, or write a generic tool that can digest serialized Thrift for
some reason.

The bottom line is that the way it works now, nothing is implied, which is
probably suitable for most applications. There's certainly the possibility
that for other applications, this doesn't make as much sense, so perhaps we
should explore those ideas more fully, but *definitely* in another thread
than this one.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message