thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruce Simpson <...@incunabulum.net>
Subject C++ multiple service support; split up stub generation
Date Sun, 13 Sep 2009 00:16:41 GMT
Mark Slee wrote:
> Yeah, I would also be against making SCons our primary build system. I've also had some
very painful experiences with doing anything beyond the very basics with SCons that have cost
me many hours.
>   

    My main motivation for working on the linkage, in the beginning, was 
to get my head into the Thrift code. I've since started hacking on it a 
little anyway, read through most of the C++ library, and got a general 
picture of how it could replace XORP's XRL.

    As part of that I've submitted a very simple patch for JIRA 570 
(dupe methods):-
        http://issues.apache.org/jira/browse/THRIFT-570

    There are a number of changes we'd need to make to support fully 
async client invocation, possibly building on Rush Manbert's work in the 
process.
    I quickly found that hacking SCons isn't needed to get up and 
running with those changes first of all, so I've backed off on that for now.
    However, I may need to revisit later. It's likely that I can do what 
I need to do without modifying the Thrift C++ libraries as they ship in 
Thrift itself.

    So, I've crafted a few patches to t_cpp_generator.cc to support 
doing a few things now.  Any changes I've made default to off for normal 
'thrift' invocation, so that it doesn't violate POLA for other users:

 * generate *only* server-side or client-side stubs, allowing separate 
output directories;
 * split the server-side stub generation into a separate set of files;
 * rename the suffixes of the generated C++ files.

> One of these experiences, in fact, was trying to get SCons to handle generated thrift
files properly -- it was extraordinarily difficult, because the dependency system employed
by SCons is entirely based upon filenames. 

    We found this was an issue even with the XORP XRL generation, which 
is very similar to Thrift in intent, until we wrapped the generators in 
appropriate SCons 'Builder' targets. Same issue with all 'Jam' tools 
too, and other build languages. SCons being implemented in Python just 
made it much easier.

    [AFAIK the Boost guys are still kind of struggling with 
Boost.BuildV2; they have the problem that it's its own darned thing, and 
nobody but the authors understand it. FWIW it looks like Thrift isn't 
using much of Boost beyond shared_ptr at the moment, which is a header 
template w/o specialization in a library, so linkage isn't an issue here.]

    We end up having to ensure the generated file always appears in a 
known location. The wrappers then know enough for dependencies to be 
instantiated in the build graph OK.
    The file suffix is one point, as suffix rules can be used to build 
stubs for the Make targets, and then set up dependencies on the 
'*.thrift' service definitions themselves -- independent of the build 
language.
    In our tree it would be nice, to keep the file extensions 
consistent, to allow for tricks like this, so I've been hacking on 
changing the '.cpp' / '.h' in t_cpp_generator.cc for each service 
invocation.

    The other thing is: output directories. What we'd like to do in our 
tree, is to generate the client and server side stubs separately, so we 
can keep the server-side code in the subdirectories for the processes 
themselves.

    Multiple service support is also a bit of an issue for us just now, 
as the existing XRL code expects to be able to use the same endpoint to 
talk to multiple services in the same process.
    Implementing an adaptor like TFramedTransport looks like the way to 
go here, however, it would require a TServer instance which is aware of 
the service name in the framing, and which can dispatch using a map of 
TProcessors.

    Because of this, it's much easier for us to have those half of the 
stubs in another set of files. At the moment the server-side glue goes 
into <service>.h.
    With my patch, the 'split_processor' argument in generator_strings 
puts it into <service>_proc.h/cpp.
    Of course the server-side stubs partly depend on what goes into the 
client stubs; so we may need to be a little smarter about how we split 
the code up for linkage in our own processes.

> Would also note that SCons has very limited/poor support for lots of the languages that
Thrift employs, and as David mentions, extending SCons requires significant time and knowledge.
>   

   You're quite right that it may not be the right answer for Thrift as 
a project just now. It seems to work for XORP quite well, as it is 
almost completely C++, and there are cross directory shared library 
dependencies which Automake did not cope with very well.

    Rush's patches change a lot of stuff, and I guess I just want to be 
sure that everything's tight, particularly in a dynamic executable.
    Having said that, it's looking like going to ASIO right away is too 
expensive a change to make for XORP, and a real deployment of Thrift in 
that project probably means writing our own TServer class to integrate 
with the existing EventLoop.

cheers,
BMS

P.S. Now that I've got my head around the C++ library, building an AMQP 
adaptor doesn't look so bad. Esteve has blazed the trail in a 
framework/language which actually has an I/O model...

Mime
View raw message