thrift-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: Using thrift as part of a game network protocol
Date Fri, 03 Apr 2009 00:13:37 GMT
Could this be made less dependent on the world supply of good-will and
optimism?

That is, could there actually be a fully operational Thrift transport that
moves requests both ways on a single socket?  You would give it something
that implements the server interface and it would give you back something
that also implements the interface.  You could call it and it could call you
in horrifically complicated but very cool ways.

*That* would be something that other RPC packages don't do.

On Thu, Apr 2, 2009 at 4:56 PM, David Reiss <dreiss@facebook.com> wrote:

> What an ingenious hack!  Cross-posting to thrift-dev in case some
> people don't read this list.
>
> --David
>
> Joel Meyer wrote:
> > On Tue, Mar 24, 2009 at 5:01 PM, Doug Daniels <daniels.douglas@gmail.com
> >wrote:
> >
> >> Ok I definitely plan on giving the Async RPC methods a try tonight, but
> I
> >> figured I'd just throw out some questions before I get home to start
> >> hacking
> >> on this stuff.
> >>
> >> The one-to-one message to RPC call Async solution will let a client send
> >> messages of any given type in my defined protocol, but how would a
> server
> >> respond to a client with a message that the client didn't request? For
> >> example say I was trying to write a FPS like Quake and I want to server
> to
> >> send position updates for all clients to all clients, how would i model
> >> that
> >> as a client RPC request for that. With the Async RPC solutions I could
> make
> >> a RPC call for Map<Integer, Position> getPositionUpdates(), Now say that
> >> the
> >> client needs to request 50 other messages to be notified of. I guess the
> >> solution would be to make an Async RPC call requesting those updates and
> >> respond to it when I receive it asynchronously and then reissue another
> >> Async RPC call for the next set of updates. It just seems inefficient to
> >> actively make the client request for data when the server could
> implicitly
> >> know that when connected on this game protocol I can just send these
> >> messages to the clients without them asking for it. Not to mention you'd
> >> have make sure you don't "miss" sending a client a message if they
> finished
> >> their Async call but haven't reestablished a new one.
> >>
> >
> > I think I've done something similar to what you're trying to do, and as
> long
> > as you can commit to using only async messages it's possible to pull it
> off
> > without having to start a server on the client to accept RPCs from the
> > server.
> >
> > When your RPC is marked as async the server doesn't send a response and
> the
> > client doesn't try to read one. So, if all your RPC calls from the client
> to
> > the server are async you have effectively freed up the inbound half of
> the
> > socket connection. That means that you can use it for receiving async
> > messages from the server - the only catch is that you have to start a new
> > thread to read and dispatch the incoming async RPC calls.
> >
> > In a typical Thrift RPC system you'd create a MyService.Processor on your
> > server and a MyService.Client on your client. To do bidirectional async
> > message sending you'll need to go a step further and create a
> > MyService.Client on your server for each client that connects (this can
> be
> > accomplished by providing your own TProcessorFactory) and then on each
> > client you create a MyService.Processor. (This assumes that you've gone
> with
> > a generic MyService definition like you described above that has a bunch
> of
> > optional messages, another option would be to define separate service
> > definitions for the client and server.) With two clients connected the
> > objects in existence would look something like this:
> >
> > Server:
> >   MyService.Processor mainProcessor - handles incoming async RPCs
> >   MyService.Client clientA - used to send outgoing async RPCs to ClientA
> >   MyService.Client clientB - used to send outgoing async RPCs to ClientB
> >
> > ClientA:
> >   MyService.Client - used to send messages to Server
> >   MyService.Processor clientProcessor - used (by a separate thread) to
> > process incoming async RPCs
> >
> > ClientB:
> >   MyService.Client - used to send messages to Server
> >   MyService.Processor clientProcessor - used (by a separate thread) to
> > process incoming async RPCs
> >
> > Hopefully that explains the concept. If you need example code I can try
> and
> > pull something together (it will be in Java). The nice thing about this
> > method is that you don't have to establish two connections, so you can
> get
> > around the firewall issues others have mentioned. I've been using this
> > method on a service in production and haven't had any problems. When you
> > have a separate thread in your client running a Processor you're
> basically
> > blocking on a read, waiting for a message from the server. The benefit of
> > this is that you're notified immediately when the server shuts down
> instead
> > of having to wait until you send a message and then finding out that the
> TCP
> > connection was reset.
> >
> > Cheers,
> > Joel
> >
> >
> >> The biggest issue is that not all client request will result in a single
> >> response (like shooting a bullet, may blowup an entity, and damage all
> >> players in the area those events are seperate messages sent from the
> >> respective entities).
> >>
> >> At a game development studio I used to work at we developed a cross
> >> language
> >> IDL network protocol definition (C++, Java)  very similiar to Protocol
> >> Buffers and Thrift (without some of the more mature features like being
> >> transport agnostic we explicitly built it for binary TCP socket
> transport,
> >> or protocol versioning), the stream of packets would contain as the
> first
> >> 32
> >> bits a message ID that would be a key to a map a Message class that
> would
> >> have methods to read in that message type from a byte[] stream.
> >>
> >> Looking through Thrift code in the TBinaryProtocol writeMessage it looks
> >> like it's including the name of the message being sent and it's type (is
> >> the
> >> concept of Message in thrift the same as RPC?), if so what's the
> >> corresponding code pathway for the client waiting for an RPC response
> >> because if I could just use this message name or type to key into what I
> >> need to serialize off the network from both client and server end then
> that
> >> would be perfect.
> >>
> >>
> >>
> >> On Tue, Mar 24, 2009 at 1:51 PM, Ted Dunning <ted.dunning@gmail.com>
> >> wrote:
> >>
> >>> I really think that using async service methods which are matched one
> to
> >>> one
> >>> with the message types that you want to send gives you exactly the
> >>> semantics
> >>> that are being requested with very simple implementation cost.
> >>>
> >>> It is important to not get toooo hung up on what RPC stands for.  I use
> >>> async methods all the time to stream data structures for logging and it
> >>> works great.  Moreover, it provides a really simple way of building
> >>> extractors and processors for this data since I have an interface
> sitting
> >>> there that will tell me about all of the methods (data types) that I
> need
> >>> to
> >>> handle or explicitly ignore.
> >>>
> >>> So the trick works and works really well.  Give it a try!
> >>>
> >>> On Tue, Mar 24, 2009 at 8:23 AM, Bryan Duxbury <bryan@rapleaf.com>
> >> wrote:
> >>>> Optional fields are not serialized onto the wire. There is a slight
> >>>> performance penalty at serialization time if you have a ton of unset
> >>> fields,
> >>>> but that's it.
> >>>>
> >>>>  Am I over complicating things
> >>>> Personally, sounds like it to me. Why do you need this streaming
> >> behavior
> >>>> or whatnot? Hotwiring the rpc stack to let you send any message you
> >> want
> >>> is
> >>>> going to be a ton of work and not really that much of a functionality
> >>>> improvement.
> >>>>
> >>>> -Bryan
> >>>>
> >>>
> >>>
> >>> --
> >>> Ted Dunning, CTO
> >>> DeepDyve
> >>>
>



-- 
Ted Dunning, CTO
DeepDyve

111 West Evelyn Ave. Ste. 202
Sunnyvale, CA 94086
www.deepdyve.com
858-414-0013 (m)
408-773-0220 (fax)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message