thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mayan Moudgill <ma...@bestweb.net>
Subject Streaming
Date Fri, 30 Apr 2010 14:06:05 GMT

I've implemented a version of client-side streaming for lists in 
cthrift. It is _fully_ compatible with the Thrift TBinaryProtocol. It 
should be possible to adapt these ideas to Thrift.

It works as follows:
1. Split the RPC into a send and a recieve part, so instead of
   foo()
  call
    foo_send()
    foo_recv()

2. For sending a stream, split into the part before the streamed list 
(including the count), a part that can send portions of the streamed 
list, and the part after the list. Thus
   foo_send_stream_start(..., len_of_list)
   while( !done ) {
     foo_send_stream_do_stream(..., len_of_chunk, chunk);
   }
   foo_send_stream_end(...)
This idea can be extended to handle multiple streams

3. For receiving a stream, the _recv() is split into two functions: the 
first which returns the count, and the other whcih returns sections of 
the stream.
   total = foo_recv_stream_len()
   n = 0;
   while( n != total ) {
     size = foo_recv_stream_data(..., max, res);
     n += size;
   }
Note that max is the maximum chunk size, while size is the size actually 
read. This can be less than max so that it can handle the case that only 
part of the stream has been transferred.


Mime
View raw message