On Thu, Oct 14, 2010 at 6:28 AM, Andrew Stitcher <astitcher@redhat.com> wrote:
> On Wed, 2010-10-13 at 21:59 -0700, Aaron Fabbri wrote:
>> On Tue, Oct 12, 2010 at 10:22 AM, Andrew Stitcher <astitcher@redhat.com> wrote:
>> > For those interested in the Rdma implementation:
>> >
>> > I've been doing a lot of stability work, stressing the rdma code in odd
>> > corner cases (unexpected disconnects mostly). While on this trail I
>> > reailised I could simplify the Rdma::AsynchIO state machine drastically
>> > by ensuring that all callbacks generated by this layer happen in the
>> > "thread context" of the connection.
>>
>> Thanks for the heads up. I'm taking a quick look at the diffs. By
>> "thread context of the connection", do you mean always having these
>> callbacks happen from the poller threads?
>
> yes.
>
>>
>> Can you give some hints on how this simplified things?
>
> Look at the code, and you will see ;-)
Fair enough. I did look at the diffs and the new stuff is much
cleaner. I was fishing for some color or background on the previous
complex state machine, and why restricting thread contexts simplifies
it so much. I'm sure I could figure it out with more than 15 minutes
staring at 1500 lines of diffs.
> In a little more detail - the Rdma::AsynchIO code is a lot easier to
> understand IMO. being sure that all the callbacks happens from an IO
> thread with the connection properly serialised makes it easier to not
> screw up changes too.
>
>>
>> >
>> > After an iteration to improve the performance which added a simple
>> > version of the state machine back, we have a version that has very
>> > similar throughput, but a little better latency as measured by me on my
>> > development boxes.
>>
>> What sort of latency improvement are you seeing?
>
> Irrelevant I think given I did no real tuning and this seems to be
> important to get good and reliable results. I was only testing in order
> to be sure there was no obvious regression.
It is curious that forcing a context switch in the write path
(notifyPendingWrite now wakes up a poller which does the idle callback
which enqueues the write) is OK performance-wise. A major motivation
of verbs/RDMA is to avoid context switches.
That's the feedback that came to mind. I know sending a patch would
be more constructive. ;-) I will give this latest stuff a go and let
you know if anything interesting happens.
Thanks,
Aaron
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
|