qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Stitcher <astitc...@redhat.com>
Subject Re: Heads up re Rdma IO state transitions [Was: AsynchIO state transition]
Date Fri, 15 Oct 2010 15:07:23 GMT
On Thu, 2010-10-14 at 15:03 -0700, Aaron Fabbri wrote:
> On Thu, Oct 14, 2010 at 6:28 AM, Andrew Stitcher <astitcher@redhat.com> wrote:
> > On Wed, 2010-10-13 at 21:59 -0700, Aaron Fabbri wrote:
> >> On Tue, Oct 12, 2010 at 10:22 AM, Andrew Stitcher <astitcher@redhat.com>
> >> > For those interested in the Rdma implementation:
> >> >
> >> > I've been doing a lot of stability work, stressing the rdma code in odd
> >> > corner cases (unexpected disconnects mostly). While on this trail I
> >> > reailised I could simplify the Rdma::AsynchIO state machine drastically
> >> > by ensuring that all callbacks generated by this layer happen in the
> >> > "thread context" of the connection.
> >>
> >> Thanks for the heads up.  I'm taking a quick look at the diffs.  By
> >> "thread context of the connection", do you mean always having these
> >> callbacks happen from the poller threads?
> >
> > yes.
> >
> >>
> >> Can you give some hints on how this simplified things?
> >
> > Look at the code, and you will see ;-)
> Fair enough. I did look at the diffs and the new stuff is much
> cleaner.  I was fishing for some color or background on the previous
> complex state machine, and why restricting thread contexts simplifies
> it so much. I'm sure I could figure it out with more than 15 minutes
> staring at 1500 lines of diffs.

The state machine changes are all in the last 2 RDMA changes checked in
(much fewer than 1500 lines of changes!). I use a git mirror of the svn
repo for all my development and find gitk a great way to inspect tree
changes. I recommend it. 

The other changes don't change the state machine at all, but do
ultimately tidy up some of the other code I think. Note that I try
fairly  hard to make each changeset coherent in itself (I go back and
alter the changesets if necessary before finally pushing them up to svn)
So you should be able to look at each change in some isolation.

> ...
> It is curious that forcing a context switch in the write path
> (notifyPendingWrite now wakes up a poller which does the idle callback
> which enqueues the write) is OK performance-wise.  A major motivation
> of verbs/RDMA is to avoid context switches.

Yes this is an interesting aspect of the code. What happens is that the
code actually avoids context switching when under load. Actually the
whole purpose of the new state machine is to this: Essentially if you
get a notifyWrite event whilst processing incoming completions then the
notifying thread just sets state so that the already processing thread
will go and do the notify callback.

The only time when you get actual context switch in the write path is
when there isn't a lot of load.

Actually this was the major purpose of the previous difficult to
understand state machine too.

What seems to be the case from my rough measurements is that calling
back on whatever thread got the notifyPendingWrite() isn't in itself a
major performance gain over hijacking another thread already servicing
the connection.

Obviously proper measurement and experience will show in the end.


Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

View raw message