thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruce Simpson <>
Subject Re: Non-blocking C++ client?
Date Sat, 20 Jun 2009 13:34:23 GMT
Hi David,

Thanks for your insightful and quick reply... I'm very optimistic about 
Thrift's healthy development community!
The fact that have chosen Thrift for infrastructure puts it 
firmly in the Venn set of 'cool kids'.

Bit of a long one. Fairly in depth technical analysis, and offer of 
support, follows...

David Reiss wrote:
> First, a quick note about the non-blocking server.  It uses nonblocking
> I/O, but the actual application callback is called synchronously
> (either in the main thread or in a worker thread), so it has to be
> synchronous (put another way, the return value for the RPC must be
> built before the handler function returns).

Yep, saw *some* of that, but not the critical observation that the 
return value must be synchronous. Thanks for pointing that one out right 

We'd have to fork TNonblockingServer anyway to integrate it into the 
XORP EventLoop.  So 'some assembly required'.
That's fine -- XRL has been enough of a sticking point anyway that 
radical change is still needed.

We don't use libevent, but rather our own stuff. Some background to why 
we do that:
    I was funded to port XORP to Microsoft Windows a few years ago, and 
this was mostly a success, in that we demonstrated Windows Server 2003 
in Redmond handling a full BGP routing table. Sadly this is now 
bitrotting due to a lack of further interest and attention from them... 
integrating with their IP forwarding requires all sorts of shim DLLs and 
service magick.
    And imagine my surprise and dismay when I learn that their augmented 
POSIX stuff, Subsystem for UNIX Applications (SUA), is about to be axed. 
The lack of attention to tools is a Development Limiting Move, to my mind.
    EventLoop is a critical path in the XORP framework. It was possible 
to leverage the existing design (tied to non-blocking BSD socket I/O) 
with a number of stealthy workarounds, without threading the entire code 
base, as we rely on the OS's socket buffer behaviour for serialization. 
Really, ioctlsocket() with FIONREAD -- and some knowledge about how 
Winsock operates in terms of helper thread context switch, which only 
became visible under careful observation with Mark Russinovich's Process 
Explorer. They don't tell you any of this in the docs -- they expect you 
to use Win32 threads.

I daresay I need to go off and look at Boost ASIO now to see if someone 
else has done similar.

> There is currently no explicit support for doing non-blocking RPC
> in Thrift (at least from C++.  I'm not sure how the Twisted stuff
> works).  Someone at Facebook is working on support for this, and
> we should be able to release it soonish.

Fingers crossed! Rough guesstimates on timescale would be great.

    I do most of my development on FreeBSD-STABLE. If there's anything I 
can do to help, please do let me know.
    I know for a fact that the FreeBSD port is bitrotted -- it is still 
shipping the code with 'facebook::' namespace prior to Apache 
Incubation, so that is definitely something I can look at in the meantime.

    My understanding is that the Thrift code still uses Boost. If that 
is still the case, great. There are things we need to do there anyway -- 
there are a whole bunch of places where XORP's idioms could be 
refactored to use Boost.

    XORP has been going as a project for so long (2002), that it's worth 
remembering we had to invent a lot of our own stuff, which is probably 
better supported now by infrastructure such as Boost and Thrift, which 
have more flexible and community-supported idioms. Also, thread 
libraries weren't as mature then as they are now.

It is also encouraging that Boost, or parts of it, are making their way 
into the ISO C++0x specification. That's what we like -- development 
must be sustainable.

>   In the mean time, one thing
> you can do is send your message to a TMemoryBuffer, transport that
> to the backend using whatever system you want, then receive the
> result from another TMemoryBuffer. 

OK. I'll transliterate into XORP-speak here for adaptation (mostly for 
my own reference, but, it may interest people...)
looking at

>  For example, if you have a function
> called "foo":
> shared_ptr<TMemoryBuffer> buf(new TMemoryBuffer);

Analogous (without transport hooks) to re-using existing XrlRouter 
reference when instantiating Xrl client stub.
e.g. _xrl_router.

(An XrlRouter is an object used to connect to the Finder, XORP XRL's 
request broker.)

> shared_ptr<MyClient> cl(new MyClient(shared_ptr<TProtocol>(new TBinaryProtocol(buf)));
Analogous to instantiating Xrl generated client stub e.g.

	XrlRawPacket4V0p1Client fea_client(&_xrl_router);

> cl->send_foo(args);

Analogous to calling fea_client.send_foo(...), although, WITHOUT the 
callback binding.
We use member function pointers a lot.

> uint8_t* bytes;
> uint32_t nbytes;
> buf->getBuffer(&bytes, &nbytes);
> // stash buf and cl somewhere.

This is what the  XrlRawPacket4V0p1Client._sender instance of XrlSender 
is going to be doing.
It binds the callback passed in -- and invokes it when the EventLoop 
(Reactor pattern) gets activity on the socket(s) used for XRL transport.

> // ship nbytes of bytes off to the server

OK. Understand we need to do our own transport stuff here;
understand that TNonblockingServer comes with limitations for transport 
used anyway.

> // get the server's response
> buf->resetBuffer(response, response_len);
> int ret = cl->recv_foo();

This is surprisingly similar to what the XRL client glue does for 
marshaling the reply and then dispatching the callback.

I still have some unanswered questions about what we'd do about 
replacing the Finder. As I see it, Thrift in its default incarnation 
leaves the specifics of transport endpoint up to each application (based 
on my reading the simple C++ client/service example in the Thrift SVN 
tree), and does not present an ORB as such.

This is analogous to XRL's STCP (*not* SCTP) without service 
registration/advertisement. AMQP would most likely deal with some of 
that for us -- but in the absence of such integration, we'd need a 
lightweight broker mechanism which would work with our chosen 
transports. I do wonder if someone has already solved that problem 
in/around Thrift.

I noticed that there's a patch for dealing with multiple Thrift servers 
here (may be bit-rotted, looks as though it's for the pre-Apache tree):-

We'd probably need something like this. At the moment, the XRL framework 
takes multiple XIF definitions and spits out bindings coalesced into one 
'target' containing every XIF interface implemented for the software 
component in a single C++ class. The methods are abstract virtual so the 
class needs to be derived from in order to make sure all the bindings 
are implemented for each XRL target (analogy: collection of thrift 

Obviously Thrift doesn't do this -- C++ bindings are generated for each 
server component -- so -- we would need to be able to tie in multiple 
Thrift server-side stubs into the same component. But this is something 
we can revisit as we figure out the integration.

One of the questions I can see people will have is: will the Thrift 
solution perform slower than the XRL solution? At the moment my gut 
feeling is that Thrift is going to be faster at binary marshaling, and 
there are problems in XORP's scalability which are better solved using 
alternate mechanisms i.e. shared memory arenas and/or batch-mode 
messaging for virtualized clusters (AMQP beckons...).

Anyway, more points about where I'm coming from:

    I'll keep my ears to the ground meanwhile with AMQP. It shouldn't be 
too much hassle to adapt what the txAMQP dude has done for Twisted 
Python -- although they have the benefit of fantastic language support 
for continuations (yield).

    Running Thrift RPC as an AMQP application strikes me as a major step 
forward for clustering and virtualization.
    We have been looking at stuff like Concurrent C++, but it requires 
very ambitious tweaks, essentially it ships its own userland scheduler 
-- and that is still subject to stuff like timer aliasing between 
userland and kernel. It also isn't sufficiently mature in terms of user 
base at the moment.
    FreeBSD's KSE M:N threading support would be fantastic here, but 
it's currently in a questionable state -- development inertia has set 
in, and folk have focused on 1:1 threading as that's what has taken hold.

More as it happens...

thanks muchly,

P.S. Thanks to Esteve for his reply on current developments in Boost. 
ASIO isn't covered in my current Boost book which is most likely in need 
of an update. ASIO probably captures some of what's already in libxorp, 
open to new ways and improvement.
P.P.S. Thank you for kicking CORBA in the crotch. Much love.

View raw message