qpid-proton mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Garlapati Sreeram Kumar <sreer...@live.com>
Subject RE: Proton-j Reactor - Receiver
Date Fri, 15 Apr 2016 00:35:54 GMT
Hello Robbie,

After couple hours of Stress Run with the proton-j change – we still could repro the Receive
Stuck problem. Although – the below bug fix places us in a much better state now.
Attached a screenshot of Objects sizes on Heap – which corresponds to the codepath that
you fixed (SimpleSslTransportWrapper).
Please see if this rings any bells – I am more than happy to share more details (proton
traces & dump – pls suggest if you will need more details).

Thx!
Sree

From: Garlapati Sreeram Kumar<mailto:sreeramg@live.com>
Sent: Monday, April 11, 2016 11:50 AM
To: Robbie Gemmell<mailto:robbie.gemmell@gmail.com>; proton@qpid.apache.org<mailto:proton@qpid.apache.org>
Cc: SeongJoon Kwak (SJ)<mailto:sjkwak@microsoft.com>; hmlam@microsoft.com<mailto:hmlam@microsoft.com>
Subject: RE: Proton-j Reactor - Receiver

Awesome.

To make it easy - added you as collaborator to my fork of Proton & here’s the branch
from which I submitted the PR: https://github.com/SreeramGarlapati/qpid-proton/tree/sg.recvstuck

Thx!
Sree

From: Robbie Gemmell<mailto:robbie.gemmell@gmail.com>
Sent: Monday, April 11, 2016 9:52 AM
To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
Cc: SeongJoon Kwak (SJ)<mailto:sjkwak@microsoft.com>; hmlam@microsoft.com<mailto:hmlam@microsoft.com>
Subject: Re: Proton-j Reactor - Receiver

Ah, excellent. I had actually started on testing this myself a little
earlier, so I'll take a look and see whats what before continuing
tomorrow. On taking an initial better look at things I think the
change itself may need augmented to account for some other conditions
too, need to investigate further to be sure.

Robbie

On 11 April 2016 at 17:37, Garlapati Sreeram Kumar <sreeramg@live.com> wrote:
> Thanks a lot for the Response Robbie!
> Per your suggestion, added the CIT to the Pull Request (& yes, as you already said
– this issue is being tracked via JIRA - PROTON-1171).
>
> Thanks a lot for the Wonderful Collaboration!
> Sree
>
> From: Robbie Gemmell<mailto:robbie.gemmell@gmail.com>
> Sent: Thursday, April 7, 2016 3:52 AM
> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
> Cc: SeongJoon Kwak (SJ)<mailto:sjkwak@microsoft.com>; hmlam@microsoft.com<mailto:hmlam@microsoft.com>
> Subject: Re: Proton-j Reactor - Receiver
>
> Hi Sree,
>
> Thanks for the analysis and PR, I'll try to take a proper look soon.
> It's not an area of the code I'm familiar with so I'll need to have a
> bit of a dig myself to see if the change seems ok. I'd note that any
> not-insignificant bug fix such as this should probably have a test
> with it (and a JIRA, though I see you have since created one of those)
> :)
>
> Robbie
>
> On 6 April 2016 at 01:23, Garlapati Sreeram Kumar <sreeramg@live.com> wrote:
>> Hello Robbie,
>>
>> We are using proton-j client with SSL and many of our customers are hitting this
issue.
>> Here are my findings after debugging through this issue:
>>
>> -          When incoming bytes arrive on the SocketChannel – proton-j client gets
signaled by nio & as a result it unwinds the transport stack – as a result all the TransportInput
implementations performs its task on the Read Bytes and hands off to the Next Layer in the
stack (transport to ssl, ssl to frameparser etc).
>>
>> -          While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput reads(16k
bytes) from _inputBuffer and the result - decoded bytes are written to _decodedInputBuffer
– as an intermediate buffer.
>>
>> -          It then flushes bytes from intermediate buffer to the next layer &
invokes an _underlyingInput.Process() – to signal it that it has bytes in its input buffer.
>>
>> -          If the underlyingInput (lets say FrameParser) buffer size is small –
lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over time this accrues.
>>
>> The fix here is to flush decodedInputBuffer to the Next transport in the Network
Stack & call _underlyingInput.Process() - until decodedInputBuffer is empty. Here’s
the pull request - https://github.com/apache/qpid-proton/pull/73
>>
>> Pl. let me know if we need to do more to fix this issue comprehensively.
>>
>> Thx!
>> Sree
>>
>> From: Robbie Gemmell<mailto:robbie.gemmell@gmail.com>
>> Sent: Thursday, March 31, 2016 9:19 AM
>> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org>
>> Subject: Re: Proton-j Reactor - Receiver
>>
>> On 31 March 2016 at 04:32, Garlapati Sreeram Kumar <sreeramg@live.com> wrote:
>>> Hello All!
>>>
>>> I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP Messages
(from Microsoft Azure Event Hubs): https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124
>>>
>>> Am using the onDelivery(Event) callback to receive messages. I really appreciate
your help with this issue/behavior:
>>>
>>> ISSUE: I noticed that the last few messages on the Queue are not being issued
to onDelivery(Event) callback by the Reactor
>>> - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and discovered
that the Transfer frames corresponding to those messages were not even delivered to Client.
Then, I looked at our Service Proton Frames and can clearly see that they are being delivered
by the Service. And other AMQP clients (for ex: .net client can see the Transfer frames)
>>> - Is this a known behavior?
>>> Does Reactor code path disable Nagle on underlying socket – could this be related?
or is there any other Configuration that we should be setting to see all Transfer frames received
on the Socket?
>>>
>>> Please advice.
>>>
>>> Thanks a lot in Advance!
>>> Sree
>>>
>>> Sent from Mail for Windows 10
>>>
>>
>> I'm not aware of anyone else reporting anything like that. I don't see
>> anything in the code suggesting the reactor sets TCP_NODELAY trueon
>> the socket, but I wouldn't think that should matter here.
>>
>> The frame trace logging is done after the bytes are given to the
>> Transport and are processed into frames, so a lack of logging could
>> suggest various things such as they didnt actually get there, they
>> werent processed, something went wrong before they did/were, something
>> went wrong decoding them, etc. Its hard to say much more without more
>> info.
>>
>> Robbie

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message