thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Budric Bundy <bud...@gmail.com>
Subject Re: streaming binary data
Date Tue, 17 Dec 2013 17:07:56 GMT
Thanks Ben,

I dug through the code and I can't find the TSocket information you
mentioned.  TConnectionInfo is defined as follows in TProcessor.h:
struct TConnectionInfo {
  // The input and output protocols
  boost::shared_ptr<protocol::TProtocol> input;
  boost::shared_ptr<protocol::TProtocol> output;
  // The underlying transport used for the connection
  // This is the transport that was returned by TServerTransport::accept(),
  // and it may be different than the transport pointed to by the input and
  // output protocols.
  boost::shared_ptr<transport::TTransport> transport;
};

TTransport doesn't have TSocket or getPeer*(), and neither does TProtocol.
 I'm using thrift 0.9.1.  I'm also using TThreadPoolServer which also has
no accessor methods to socket.

Thanks.


On Tue, Dec 17, 2013 at 9:32 AM, Ben Craig <ben.craig@ni.com> wrote:

> > What I mean - unless I'm missing something - an interface like this:
> > void send(1: binary data);
> >
> > would mean the client would load all the data into memory std::string
> > object before making the call to the server.  The server would receive
> all
> > the binary data before actually calling my implementation to process the
> > data - and this is much worse, because 5-10 clients could crash the
> server
> > with out of memory error.
> >
> > Is this assumption correct?  Or is there a way to make it work?
>
> Your assumption is correct.  Current C++ client and server implementations
> only operate on full messages.  However, if you use a TFramedTransport and
> the TNonBlockingServer, you can set a maximum frame size.  You can then
> tell have your legitimate clients send megabytes at a time, in multiple
> messages.
>
> > My solution would be to have the client act as a server and use
> callback.
> >  So the interface would be
> > //Client calls server startSend()
> > void startSend(1: string callbackHost, 2: callbackPort, 3: int64
> numChunks);
> > //Server calls client host/port getChunk()
> > binary getChunk(1: int64 chunkIndex);
>
> The "getChunk" method has the same basic problem as "send(1: binary
> data)".  The server has to read the entirety of the return value before
> doing any processing.  You do get a bit more control on how many
> "getChunk" calls are outstanding, but message size capping still needs to
> happen.
>
> > The small issue I'm having is figuring out the host/port to send.  I
> can't
> > find anywhere in Thrift API where I can query the IP address and port
> the
> > thrift server is running on.  I want to get that socket information.
> There
> > may be multiple interfaces on the system, and I don't want to query the
> OS
> > for IP addresses.
>
> I'm not sure what your control flow looks like here.  To avoid some
> confusion I will refer to the "client" as Alice, and the "server" as Bob.
> Further usages of the term client and server will refer to Thrift clients
> and servers.
>
> Bob implements an IfFactory similar to the following:
>
> class arithmeticIfCloneFactory : public arithmetic::arithmeticIfFactory {
>  public:
>   virtual ~arithmeticIfCloneFactory();
>   virtual arithmetic::arithmeticIf* getHandler(const
> ::apache::thrift::TConnectionInfo& connInfo);
>   virtual void releaseHandler(arithmetic::arithmeticIf* handler);
> };
>
>
> Note the call getHandler.  The TConnectionInfo object is handed your
> IfFactory whenever a client connects.  So when Alice connects to Bob, Bob
> can get IP information about Alice (look at the getPeer* methods on
> TSocket).  Alice should then send a message telling Bob which port Bob
> should use for the reverse connection.  Bob can then use the combination
> of the connection info and Alice's message to establish a connection in
> the opposite direction.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message