mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Emmanuel Lecharny (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DIRMINA-1107) SslHandler flushScheduledEvents race condition, redux
Date Wed, 15 May 2019 14:04:00 GMT

    [ https://issues.apache.org/jira/browse/DIRMINA-1107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840433#comment-16840433
] 

Emmanuel Lecharny commented on DIRMINA-1107:
--------------------------------------------

Side note: The way encrypted messages are handled is a bit mad. Consider a BIG message (say,
1Mb). SSL does not allow messages bigger than roughly 16kb (2^14 minus header, MAC and padding.
Except for M$, of course, which allows up to 32kb messages...). In any case, we will start
with a 16Kb buffer, call encrypt again and again until the full source has been read and encrypted,
increasing the buffer as needed. This buffer will contain *many* TLS {{APPLICATION_DATA}}
records (starting with {{0x17}}). 

We could perfectly chose to always use a fixed size buffer, and once full, send it to the
remote peer, avoiding the allocation of a crazy big buffer which will be discarded when done.
The only problem is that we would need to remember when we have fully written the source,
inorder to properly send the {{messageSent}} event.

> SslHandler flushScheduledEvents race condition, redux
> -----------------------------------------------------
>
>                 Key: DIRMINA-1107
>                 URL: https://issues.apache.org/jira/browse/DIRMINA-1107
>             Project: MINA
>          Issue Type: Bug
>    Affects Versions: 2.1.2
>            Reporter: Guus der Kinderen
>            Priority: Major
>             Fix For: 2.1.3
>
>
> DIRMINA-1019 addresses a race condition in SslHandler, but unintentionally replaces it
with another multithreading issue.
> The fix for DIRMINA-1019 introduces a counter that contains the number of events to be
processed. A simplified version of the code is included below.
> {code:java}
> private final AtomicInteger scheduledEvents = new AtomicInteger(0);
> void flushScheduledEvents() {
>     scheduledEvents.incrementAndGet();
>     if (sslLock.tryLock()) {            
>         try {
>             do {
>                 while ((event = filterWriteEventQueue.poll()) != null) {
>                     // ...
>                 }
>             
>                 while ((event = messageReceivedEventQueue.poll()) != null){
>                     // ...
>                 }
>             } while (scheduledEvents.decrementAndGet() > 0);
>         } finally {
>             sslLock.unlock();
>         }
>     }
> }{code}
> We have observed occasions where the value of {{scheduledEvents}} becomes a negative
value, while at the same time {{filterWriteEventQueue}} go unprocessed.
> We suspect that this issue is triggered by a concurrency issue caused by the first thread
decrementing the counter after a second thread incremented it, but before it attempted to
acquire the lock.
> This allows the the first thread to empty the queues, decrementing the counter to zero
and release the lock, after which the second thread acquires the lock successfully. Now, the
second thread processes any elements in {{filterWriteEventQueue}}, and then processes any
elements in {{messageReceivedEventQueue}}. If in between these two checks yet another thread
adds a new element to {{filterWriteEventQueue}}, this element can go unprocessed (as the second
thread does not loop, since the counter is zero or negative, and the third thread can fail
to acquire the lock).
> It's a seemingly unlikely scenario, but we are observing the behavior when our systems
are under high load.
> We've applied a code change after which this problem is no longer observed. We've removed
the counter, and check on the size of the queues instead:
> {code:java}
> void flushScheduledEvents() {
>     if (sslLock.tryLock()) {            
>         try {
>             do {
>                 while ((event = filterWriteEventQueue.poll()) != null) {
>                     // ...
>                 }
>             
>                 while ((event = messageReceivedEventQueue.poll()) != null){
>                     // ...
>                 }
>             } while (!filterWriteEventQueue.isEmpty() || !messageReceivedEventQueue.isEmpty());
>         } finally {
>             sslLock.unlock();
>         }
>     }
> }{code}
> This code change, as illustrated above, does introduce a new potential problem. Theoretically,
an event could be added to the queues and {{flushScheduledEvents}} be called returning {{false}}
for {{sslLock.tryLock()}}, exactly after another thread just finished the {{while}} loop,
but before releasing the lock. This again would cause events to go unprocessed.
> We've not observed this problem in the wild yet, but we're uncomfortable applying this
change as-is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message