mina-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gaston Dombiak" <gas...@jivesoftware.com>
Subject SSLFilter deadlock (branch 1.1)
Date Fri, 05 Jan 2007 22:04:06 GMT
Hey,

 

I'm using an ExecutorFilter as the first filter and then an SSLFilter.
Now lets say that we have 2 clients connected to a server (that is using
MINA) and both clients are sending data to the server which in turn will
forward to the other client. With the above case I'm finding the
following deadlock in the server:

 

Found one Java-level deadlock:

=============================

"client-10":

  waiting to lock monitor 0x0bb07cd4 (object 0x0487eb70, a
org.apache.mina.filter.support.SSLHandler),

  which is held by "client-4"

"client-4":

  waiting to lock monitor 0x0bb079fc (object 0x048621f8, a
org.apache.mina.filter.support.SSLHandler),

  which is held by "client-10"

 

Java stack information for the threads listed above:

===================================================

"client-10":

            at
org.apache.mina.filter.SSLFilter.filterWrite(SSLFilter.java:455)

            - waiting to lock <0x0487eb70> (a
org.apache.mina.filter.support.SSLHandler)

            at
org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterW
rite(AbstractIoFilterChain.java:445)

            at
org.apache.mina.common.support.AbstractIoFilterChain.access$1400(Abstrac
tIoFilterChain.java:54)

            at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterW
rite(AbstractIoFilterChain.java:824)

            at
org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCod
ecFilter.java:213)

            at
org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterW
rite(AbstractIoFilterChain.java:445)

            at
org.apache.mina.common.support.AbstractIoFilterChain.access$1400(Abstrac
tIoFilterChain.java:54)

            at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterW
rite(AbstractIoFilterChain.java:824)

            at
org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.filterWr
ite(AbstractIoFilterChain.java:727)

            at
org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterW
rite(AbstractIoFilterChain.java:445)

            at
org.apache.mina.common.support.AbstractIoFilterChain.fireFilterWrite(Abs
tractIoFilterChain.java:436)

            at
org.apache.mina.transport.socket.nio.SocketSessionImpl.write0(SocketSess
ionImpl.java:197)

            at
org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:14
9)

            at
org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:13
5)

            at
org.jivesoftware.wildfire.nio.NIOConnection.deliver(NIOConnection.java:2
14)

            at
org.jivesoftware.wildfire.session.ClientSession.deliver(ClientSession.ja
va:729)

            at
org.jivesoftware.wildfire.session.Session.process(Session.java:281)

            at
org.jivesoftware.wildfire.muc.spi.MUCRoleImpl.send(MUCRoleImpl.java:241)

            at
org.jivesoftware.wildfire.muc.spi.MUCRoomImpl.broadcast(MUCRoomImpl.java
:842)

            at
org.jivesoftware.wildfire.muc.spi.MUCRoomImpl.send(MUCRoomImpl.java:772)

            at
org.jivesoftware.wildfire.muc.spi.MUCRoomImpl.sendPublicMessage(MUCRoomI
mpl.java:754)

            at
org.jivesoftware.wildfire.muc.spi.MUCUserImpl.process(MUCUserImpl.java:2
24)

            at
org.jivesoftware.wildfire.muc.spi.MUCUserImpl.process(MUCUserImpl.java:1
30)

            at
org.jivesoftware.wildfire.muc.spi.MultiUserChatServerImpl.processPacket(
MultiUserChatServerImpl.java:232)

            at
org.jivesoftware.wildfire.muc.spi.MultiUserChatServerImpl.process(MultiU
serChatServerImpl.java:213)

            at
org.jivesoftware.wildfire.MessageRouter.route(MessageRouter.java:96)

            at
org.jivesoftware.wildfire.spi.PacketRouterImpl.route(PacketRouterImpl.ja
va:71)

            at
org.jivesoftware.wildfire.net.StanzaHandler.processMessage(StanzaHandler
.java:346)

            at
org.jivesoftware.wildfire.net.ClientStanzaHandler.processMessage(ClientS
tanzaHandler.java:87)

            at
org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:2
09)

            at
org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:1
76)

            at
org.jivesoftware.wildfire.nio.ConnectionHandler.messageReceived(Connecti
onHandler.java:93)

            at
org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageR
eceived(AbstractIoFilterChain.java:703)

            at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageRece
ived(AbstractIoFilterChain.java:362)

            at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(Abstrac
tIoFilterChain.java:54)

            at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.message
Received(AbstractIoFilterChain.java:800)

            at
org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(S
impleProtocolDecoderOutput.java:62)

            at
org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(Protoco
lCodecFilter.java:186)

            at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageRece
ived(AbstractIoFilterChain.java:362)

            at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(Abstrac
tIoFilterChain.java:54)

            at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.message
Received(AbstractIoFilterChain.java:800)

            at
org.apache.mina.filter.SSLFilter.handleAppDataRead(SSLFilter.java:626)

            at
org.apache.mina.filter.SSLFilter.handleSSLData(SSLFilter.java:604)

            at
org.apache.mina.filter.SSLFilter.messageReceived(SSLFilter.java:395)

            - locked <0x048621f8> (a
org.apache.mina.filter.support.SSLHandler)

            at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageRece
ived(AbstractIoFilterChain.java:362)

            at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(Abstrac
tIoFilterChain.java:54)

            at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.message
Received(AbstractIoFilterChain.java:800)

            at
org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilt
er.java:231)

            at
org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run
(ExecutorFilter.java:286)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecuto
r.java:885)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:907)

            at java.lang.Thread.run(Thread.java:619)

"client-4":

            at
org.apache.mina.filter.SSLFilter.filterWrite(SSLFilter.java:455)

            - waiting to lock <0x048621f8> (a
org.apache.mina.filter.support.SSLHandler)

            at
org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterW
rite(AbstractIoFilterChain.java:445)

            at
org.apache.mina.common.support.AbstractIoFilterChain.access$1400(Abstrac
tIoFilterChain.java:54)

            at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterW
rite(AbstractIoFilterChain.java:824)

            at
org.apache.mina.filter.codec.ProtocolCodecFilter.filterWrite(ProtocolCod
ecFilter.java:213)

            at
org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterW
rite(AbstractIoFilterChain.java:445)

            at
org.apache.mina.common.support.AbstractIoFilterChain.access$1400(Abstrac
tIoFilterChain.java:54)

            at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.filterW
rite(AbstractIoFilterChain.java:824)

            at
org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.filterWr
ite(AbstractIoFilterChain.java:727)

            at
org.apache.mina.common.support.AbstractIoFilterChain.callPreviousFilterW
rite(AbstractIoFilterChain.java:445)

            at
org.apache.mina.common.support.AbstractIoFilterChain.fireFilterWrite(Abs
tractIoFilterChain.java:436)

            at
org.apache.mina.transport.socket.nio.SocketSessionImpl.write0(SocketSess
ionImpl.java:197)

            at
org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:14
9)

            at
org.apache.mina.common.support.BaseIoSession.write(BaseIoSession.java:13
5)

            at
org.jivesoftware.wildfire.nio.NIOConnection.deliver(NIOConnection.java:2
14)

            at
org.jivesoftware.wildfire.session.ClientSession.deliver(ClientSession.ja
va:729)

            at
org.jivesoftware.wildfire.session.Session.process(Session.java:281)

            at
org.jivesoftware.wildfire.muc.spi.MUCRoleImpl.send(MUCRoleImpl.java:241)

            at
org.jivesoftware.wildfire.muc.spi.MUCRoomImpl.broadcast(MUCRoomImpl.java
:842)

            at
org.jivesoftware.wildfire.muc.spi.MUCRoomImpl.send(MUCRoomImpl.java:772)

            at
org.jivesoftware.wildfire.muc.spi.MUCRoomImpl.sendPublicMessage(MUCRoomI
mpl.java:754)

            at
org.jivesoftware.wildfire.muc.spi.MUCUserImpl.process(MUCUserImpl.java:2
24)

            at
org.jivesoftware.wildfire.muc.spi.MUCUserImpl.process(MUCUserImpl.java:1
30)

            at
org.jivesoftware.wildfire.muc.spi.MultiUserChatServerImpl.processPacket(
MultiUserChatServerImpl.java:232)

            at
org.jivesoftware.wildfire.muc.spi.MultiUserChatServerImpl.process(MultiU
serChatServerImpl.java:213)

            at
org.jivesoftware.wildfire.MessageRouter.route(MessageRouter.java:96)

            at
org.jivesoftware.wildfire.spi.PacketRouterImpl.route(PacketRouterImpl.ja
va:71)

            at
org.jivesoftware.wildfire.net.StanzaHandler.processMessage(StanzaHandler
.java:346)

            at
org.jivesoftware.wildfire.net.ClientStanzaHandler.processMessage(ClientS
tanzaHandler.java:87)

            at
org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:2
09)

            at
org.jivesoftware.wildfire.net.StanzaHandler.process(StanzaHandler.java:1
76)

            at
org.jivesoftware.wildfire.nio.ConnectionHandler.messageReceived(Connecti
onHandler.java:93)

            at
org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageR
eceived(AbstractIoFilterChain.java:703)

            at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageRece
ived(AbstractIoFilterChain.java:362)

            at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(Abstrac
tIoFilterChain.java:54)

            at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.message
Received(AbstractIoFilterChain.java:800)

            at
org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(S
impleProtocolDecoderOutput.java:62)

            at
org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(Protoco
lCodecFilter.java:186)

            at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageRece
ived(AbstractIoFilterChain.java:362)

            at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(Abstrac
tIoFilterChain.java:54)

            at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.message
Received(AbstractIoFilterChain.java:800)

            at
org.apache.mina.filter.SSLFilter.handleAppDataRead(SSLFilter.java:626)

            at
org.apache.mina.filter.SSLFilter.handleSSLData(SSLFilter.java:604)

            at
org.apache.mina.filter.SSLFilter.messageReceived(SSLFilter.java:395)

            - locked <0x0487eb70> (a
org.apache.mina.filter.support.SSLHandler)

            at
org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageRece
ived(AbstractIoFilterChain.java:362)

            at
org.apache.mina.common.support.AbstractIoFilterChain.access$1100(Abstrac
tIoFilterChain.java:54)

            at
org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.message
Received(AbstractIoFilterChain.java:800)

            at
org.apache.mina.filter.executor.ExecutorFilter.processEvent(ExecutorFilt
er.java:231)

            at
org.apache.mina.filter.executor.ExecutorFilter$ProcessEventsRunnable.run
(ExecutorFilter.java:286)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecuto
r.java:885)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
va:907)

            at java.lang.Thread.run(Thread.java:619)

 

As you can see from the stack trace the lock in SSLHandler (of clientA)
is kept even when notifying the next filter which will eventually need
to write some data to clientB (that is in the same situation) so the
writing operation is blocked since it is trying to acquire the lock on
the already blocked SSLHandler by the other thread. This results in a
deadlock.

 

Options (?):

1)       Should I move the SSLFilter before the ExecutorFilter? It will
probably fix this deadlock but it may slow down the thread where the
SocketIOProcessor is working.

2)       Should I use another Executor Filter? So the chain of filters
would look like: ExecutorFilter --> SSLFilter --> ExecutorFilter. Sound
like an overkiller solution for this problem.

3)       Should we try modifying the SSLFilter class so that the scope
of the synchronization block on the SSLHandler is smaller? The goal,
besides decreasing overall locking time, would be to invoke the next
filter in the chain without blocking the SSLHandler.

4)       Any other options?

 

Note: I would like to proceed with option #3 but I'm not a MINA expert
so I don't know the reasons why the current solution was
selected/implemented.

 

Thanks,

 

  -- Gato

 

 

 

 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message