thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Budric Bundy <>
Subject streaming binary data
Date Mon, 16 Dec 2013 23:11:56 GMT

I'm writing C++ thrift client/server.  I'd like to stream large amounts of
binary data (gigabytes) between thrift client/server without loading all of
it into memory.

What I mean - unless I'm missing something - an interface like this:
void send(1: binary data);

would mean the client would load all the data into memory std::string
object before making the call to the server.  The server would receive all
the binary data before actually calling my implementation to process the
data - and this is much worse, because 5-10 clients could crash the server
with out of memory error.

Is this assumption correct?  Or is there a way to make it work?

My solution would be to have the client act as a server and use callback.
 So the interface would be
//Client calls server startSend()
void startSend(1: string callbackHost, 2: callbackPort, 3: int64 numChunks);
//Server calls client host/port getChunk()
binary getChunk(1: int64 chunkIndex);

The small issue I'm having is figuring out the host/port to send.  I can't
find anywhere in Thrift API where I can query the IP address and port the
thrift server is running on.  I want to get that socket information.  There
may be multiple interfaces on the system, and I don't want to query the OS
for IP addresses.

Thanks for suggestions.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message