qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Fabbri <ajfab...@gmail.com>
Subject Re: Heads up re Rdma IO state transitions [Was: AsynchIO state transition]
Date Thu, 14 Oct 2010 22:03:46 GMT
On Thu, Oct 14, 2010 at 6:28 AM, Andrew Stitcher <astitcher@redhat.com> wrote:
> On Wed, 2010-10-13 at 21:59 -0700, Aaron Fabbri wrote:
>> On Tue, Oct 12, 2010 at 10:22 AM, Andrew Stitcher <astitcher@redhat.com> wrote:
>> > For those interested in the Rdma implementation:
>> >
>> > I've been doing a lot of stability work, stressing the rdma code in odd
>> > corner cases (unexpected disconnects mostly). While on this trail I
>> > reailised I could simplify the Rdma::AsynchIO state machine drastically
>> > by ensuring that all callbacks generated by this layer happen in the
>> > "thread context" of the connection.
>>
>> Thanks for the heads up.  I'm taking a quick look at the diffs.  By
>> "thread context of the connection", do you mean always having these
>> callbacks happen from the poller threads?
>
> yes.
>
>>
>> Can you give some hints on how this simplified things?
>
> Look at the code, and you will see ;-)

Fair enough. I did look at the diffs and the new stuff is much
cleaner.  I was fishing for some color or background on the previous
complex state machine, and why restricting thread contexts simplifies
it so much. I'm sure I could figure it out with more than 15 minutes
staring at 1500 lines of diffs.

> In a little more detail - the Rdma::AsynchIO code is a lot easier to
> understand IMO. being sure that all the callbacks happens from an IO
> thread with the connection properly serialised makes it easier to not
> screw up changes too.
>
>>
>> >
>> > After an iteration to improve the performance which added a simple
>> > version of the state machine back, we have a version that has very
>> > similar throughput, but a little better latency as measured by me on my
>> > development boxes.
>>
>> What sort of latency improvement are you seeing?
>
> Irrelevant I think given I did no real tuning and this seems to be
> important to get good and reliable results. I was only testing in order
> to be sure there was no obvious regression.

It is curious that forcing a context switch in the write path
(notifyPendingWrite now wakes up a poller which does the idle callback
which enqueues the write) is OK performance-wise.  A major motivation
of verbs/RDMA is to avoid context switches.

That's the feedback that came to mind.  I know sending a patch would
be more constructive.  ;-)  I will give this latest stuff a go and let
you know if anything interesting happens.

Thanks,
Aaron

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org


Mime
View raw message