thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Craig <>
Subject Re: streaming binary data
Date Tue, 17 Dec 2013 14:32:27 GMT
> What I mean - unless I'm missing something - an interface like this:
> void send(1: binary data);
> would mean the client would load all the data into memory std::string
> object before making the call to the server.  The server would receive 
> the binary data before actually calling my implementation to process the
> data - and this is much worse, because 5-10 clients could crash the 
> with out of memory error.
> Is this assumption correct?  Or is there a way to make it work?

Your assumption is correct.  Current C++ client and server implementations 
only operate on full messages.  However, if you use a TFramedTransport and 
the TNonBlockingServer, you can set a maximum frame size.  You can then 
tell have your legitimate clients send megabytes at a time, in multiple 

> My solution would be to have the client act as a server and use 
>  So the interface would be
> //Client calls server startSend()
> void startSend(1: string callbackHost, 2: callbackPort, 3: int64 
> //Server calls client host/port getChunk()
> binary getChunk(1: int64 chunkIndex);

The "getChunk" method has the same basic problem as "send(1: binary 
data)".  The server has to read the entirety of the return value before 
doing any processing.  You do get a bit more control on how many 
"getChunk" calls are outstanding, but message size capping still needs to 

> The small issue I'm having is figuring out the host/port to send.  I 
> find anywhere in Thrift API where I can query the IP address and port 
> thrift server is running on.  I want to get that socket information. 
> may be multiple interfaces on the system, and I don't want to query the 
> for IP addresses.

I'm not sure what your control flow looks like here.  To avoid some 
confusion I will refer to the "client" as Alice, and the "server" as Bob. 
Further usages of the term client and server will refer to Thrift clients 
and servers.

Bob implements an IfFactory similar to the following:

class arithmeticIfCloneFactory : public arithmetic::arithmeticIfFactory {
  virtual ~arithmeticIfCloneFactory();
  virtual arithmetic::arithmeticIf* getHandler(const 
::apache::thrift::TConnectionInfo& connInfo);
  virtual void releaseHandler(arithmetic::arithmeticIf* handler);

Note the call getHandler.  The TConnectionInfo object is handed your 
IfFactory whenever a client connects.  So when Alice connects to Bob, Bob 
can get IP information about Alice (look at the getPeer* methods on 
TSocket).  Alice should then send a message telling Bob which port Bob 
should use for the reverse connection.  Bob can then use the combination 
of the connection info and Alice's message to establish a connection in 
the opposite direction.

View raw message