qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Stitcher <astitc...@redhat.com>
Subject Re: Lower msg latency in Qpid Rdma?
Date Mon, 27 Apr 2009 14:09:51 GMT
On Sat, 2009-04-25 at 13:24 -0400, gregory james marsh wrote:
> Andrew,
> Thanks for the response to my post.
> What would be the consequences of me not registering the FD of a 
> Rdma::AsynchIO with the Poller?
> Instead I would have Rdma::AsynchIO internally poll it's own completion 
> queue (CQ) for the first message and then call the 
> "Rdma::AsynchIO.DataEvent ->  Rdma::AsynchIO.ProcessCompletion -> 
> QueuePair.GetNextEvent" chain that the Dispatcher/Poller would call.  When 
> that's complete Rdma::AsynchIO would go back to polling for another first 
> message and so on.
> Since each Rdma::AsynchIO would be handling its own polling, message flow 
> would be serialized via the CQ.

I think this appraoch would work.

A concern is that in a broker that has multiple transports active busy
waiting such as you are describing would wastefully steal resources away
from other transports and activities.

If you only need RDMA this might not be a concern, but it doesn't sound
like a good approach for a more general broker that has to support TCP,

> You mention that the Poller "...serializes all events occurring on a given 
> registered Handle."  Out of curiosity is this being done by the scoped 
> locks on the Mutex DispatchHandle.stateLock?  (For example in 
> DispatchHandle.processEvent).

No, the guarantee is, in the Linux case, given by the semantics of the
way epoll is used. In this case we set it up so that once an event has
been triggered for an fd, events are disabled until we re-enable them
again explicitly.

If you busy loop poll the cq, I think this is likely to give the same
semantics just so long as each cq has it's own busy loop, and they are
not shared between threads.


Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

View raw message