thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Hammond <>
Subject Re: Using thrift as part of a game network protocol
Date Sat, 04 Apr 2009 01:22:52 GMT
That's neat Joel.  However, does this scale?  I mean, the underlying  
assumption here is that clients are using persistent connections to  
the service, and you [not so simply] are sending messages back to the  
client over that same connection.  Thus, your service now has to  
handle a potentially large number of client connections.  Unless  
you're using something libev[ent] I don't see this scaling beyond say  
20K connections.  Two things, I could be missing something here, and  
this level of scalability is probably just fine for *many* types of  
services (perhaps not for a chat server though)!

I'm curious to hear other people's thoughts on this and how it could  
be made scalable since, well, I'm planning on using polling in my  
project since I am expecting potentially a very large number of  
simultaneous users of the service and my servers can only handle so  
many connections.

Thanks for sharing this.


On Apr 3, 2009, at 7:50 PM, Joel Meyer wrote:

> On Thu, Apr 2, 2009 at 4:17 PM, Joel Meyer <>  
> wrote:
>> On Tue, Mar 24, 2009 at 5:01 PM, Doug Daniels < 
>> >wrote:
>>> Ok I definitely plan on giving the Async RPC methods a try  
>>> tonight, but I
>>> figured I'd just throw out some questions before I get home to start
>>> hacking
>>> on this stuff.
>>> The one-to-one message to RPC call Async solution will let a  
>>> client send
>>> messages of any given type in my defined protocol, but how would a  
>>> server
>>> respond to a client with a message that the client didn't request?  
>>> For
>>> example say I was trying to write a FPS like Quake and I want to  
>>> server to
>>> send position updates for all clients to all clients, how would i  
>>> model
>>> that
>>> as a client RPC request for that. With the Async RPC solutions I  
>>> could
>>> make
>>> a RPC call for Map<Integer, Position> getPositionUpdates(), Now  
>>> say that
>>> the
>>> client needs to request 50 other messages to be notified of. I  
>>> guess the
>>> solution would be to make an Async RPC call requesting those  
>>> updates and
>>> respond to it when I receive it asynchronously and then reissue  
>>> another
>>> Async RPC call for the next set of updates. It just seems  
>>> inefficient to
>>> actively make the client request for data when the server could  
>>> implicitly
>>> know that when connected on this game protocol I can just send these
>>> messages to the clients without them asking for it. Not to mention  
>>> you'd
>>> have make sure you don't "miss" sending a client a message if they
>>> finished
>>> their Async call but haven't reestablished a new one.
>> I think I've done something similar to what you're trying to do,  
>> and as
>> long as you can commit to using only async messages it's possible  
>> to pull it
>> off without having to start a server on the client to accept RPCs  
>> from the
>> server.
>> When your RPC is marked as async the server doesn't send a response  
>> and the
>> client doesn't try to read one. So, if all your RPC calls from the  
>> client to
>> the server are async you have effectively freed up the inbound half  
>> of the
>> socket connection. That means that you can use it for receiving async
>> messages from the server - the only catch is that you have to start  
>> a new
>> thread to read and dispatch the incoming async RPC calls.
>> In a typical Thrift RPC system you'd create a MyService.Processor  
>> on your
>> server and a MyService.Client on your client. To do bidirectional  
>> async
>> message sending you'll need to go a step further and create a
>> MyService.Client on your server for each client that connects (this  
>> can be
>> accomplished by providing your own TProcessorFactory) and then on  
>> each
>> client you create a MyService.Processor. (This assumes that you've  
>> gone with
>> a generic MyService definition like you described above that has a  
>> bunch of
>> optional messages, another option would be to define separate service
>> definitions for the client and server.) With two clients connected  
>> the
>> objects in existence would look something like this:
>> Server:
>>  MyService.Processor mainProcessor - handles incoming async RPCs
>>  MyService.Client clientA - used to send outgoing async RPCs to  
>> ClientA
>>  MyService.Client clientB - used to send outgoing async RPCs to  
>> ClientB
>> ClientA:
>>  MyService.Client - used to send messages to Server
>>  MyService.Processor clientProcessor - used (by a separate thread) to
>> process incoming async RPCs
>> ClientB:
>>  MyService.Client - used to send messages to Server
>>  MyService.Processor clientProcessor - used (by a separate thread) to
>> process incoming async RPCs
>> Hopefully that explains the concept. If you need example code I can  
>> try and
>> pull something together (it will be in Java). The nice thing about  
>> this
>> method is that you don't have to establish two connections, so you  
>> can get
>> around the firewall issues others have mentioned. I've been using  
>> this
>> method on a service in production and haven't had any problems.  
>> When you
>> have a separate thread in your client running a Processor you're  
>> basically
>> blocking on a read, waiting for a message from the server. The  
>> benefit of
>> this is that you're notified immediately when the server shuts down  
>> instead
>> of having to wait until you send a message and then finding out  
>> that the TCP
>> connection was reset.
>> Cheers,
>> Joel
> Thanks for the feedback. I've created a simple example in Java  
> demonstrating
> this in action:
> Post with a few details on the implementation:
> Please add me to the list of people who think there's value in a  
> full async
> transport that provides (optional?) synchronization at the api level  
> using
> futures/deferreds/etc.
> Cheers,
> Joel
>>> The biggest issue is that not all client request will result in a  
>>> single
>>> response (like shooting a bullet, may blowup an entity, and damage  
>>> all
>>> players in the area those events are seperate messages sent from the
>>> respective entities).
>>> At a game development studio I used to work at we developed a cross
>>> language
>>> IDL network protocol definition (C++, Java)  very similiar to  
>>> Protocol
>>> Buffers and Thrift (without some of the more mature features like  
>>> being
>>> transport agnostic we explicitly built it for binary TCP socket  
>>> transport,
>>> or protocol versioning), the stream of packets would contain as  
>>> the first
>>> 32
>>> bits a message ID that would be a key to a map a Message class  
>>> that would
>>> have methods to read in that message type from a byte[] stream.
>>> Looking through Thrift code in the TBinaryProtocol writeMessage it  
>>> looks
>>> like it's including the name of the message being sent and it's  
>>> type (is
>>> the
>>> concept of Message in thrift the same as RPC?), if so what's the
>>> corresponding code pathway for the client waiting for an RPC  
>>> response
>>> because if I could just use this message name or type to key into  
>>> what I
>>> need to serialize off the network from both client and server end  
>>> then
>>> that
>>> would be perfect.
>>> On Tue, Mar 24, 2009 at 1:51 PM, Ted Dunning <>
>>> wrote:
>>>> I really think that using async service methods which are matched  
>>>> one to
>>>> one
>>>> with the message types that you want to send gives you exactly the
>>>> semantics
>>>> that are being requested with very simple implementation cost.
>>>> It is important to not get toooo hung up on what RPC stands for.   
>>>> I use
>>>> async methods all the time to stream data structures for logging  
>>>> and it
>>>> works great.  Moreover, it provides a really simple way of building
>>>> extractors and processors for this data since I have an interface
>>> sitting
>>>> there that will tell me about all of the methods (data types)  
>>>> that I
>>> need
>>>> to
>>>> handle or explicitly ignore.
>>>> So the trick works and works really well.  Give it a try!
>>>> On Tue, Mar 24, 2009 at 8:23 AM, Bryan Duxbury <>
>>> wrote:
>>>>> Optional fields are not serialized onto the wire. There is a  
>>>>> slight
>>>>> performance penalty at serialization time if you have a ton of  
>>>>> unset
>>>> fields,
>>>>> but that's it.
>>>>> Am I over complicating things
>>>>> Personally, sounds like it to me. Why do you need this streaming
>>> behavior
>>>>> or whatnot? Hotwiring the rpc stack to let you send any message  
>>>>> you
>>> want
>>>> is
>>>>> going to be a ton of work and not really that much of a  
>>>>> functionality
>>>>> improvement.
>>>>> -Bryan
>>>> --
>>>> Ted Dunning, CTO
>>>> DeepDyve

View raw message