qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Samson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (QPID-5756) Qpid Deadlock during JMS connection close when connected to ActiveMQ
Date Sat, 10 May 2014 22:12:32 GMT

    [ https://issues.apache.org/jira/browse/QPID-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993013#comment-13993013
] 

Michael Samson commented on QPID-5756:
--------------------------------------

I wrote a standalone program to attempt Connection close() when any JMSException is received.
 I ran this test when ActiveMQ was in a low memory/OutOfMemory state and noticed both
   - deadlocks in some test runs and  
   - Qpid connection output and input handler Threads never being stopped/killed when Connection
close() was called

I created a patch to ensure that Connection close() will no longer deadlock and to ensure
all Qpid output/input threads are cleaned up.  Note that my testing was performed on an .27
Qpid build.  The patch provided here was ported to the trunk source code, but I was unable
to test due to bug QPID-5755.



> Qpid Deadlock during JMS connection close when connected to ActiveMQ
> --------------------------------------------------------------------
>
>                 Key: QPID-5756
>                 URL: https://issues.apache.org/jira/browse/QPID-5756
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Client
>    Affects Versions: 0.27
>         Environment: Qpid JMS libraries 0.27 SVN revision 1547654 with socket timeout
patch
> AMQP JMS Client was running on 64-bit Linux with JDK 1.7
> AMQP Broker was ActiveMQ 5.8 running on EC2 64-bit Linux
>            Reporter: Michael Samson
>         Attachments: QPID-5756.patch, QpidProducerTestWithClose.java, activemq.xml
>
>
> 1. While sending JMS messages to ActiveMQ broker, the ActiveMQ server threw an OutOfMemoryException.
> 2. Our code processed a JMSException and attempted to shut down the Qpid JMS connection
and this processing deadlocked. 
> 3. I took several thread dumps in succession (waiting several minutes in between) and
noticed that Qpid was deadlocked.  I even killed ActiveMQ and Qpid was still deadlocked. 
Below are the some relevant threads:
> "Thread-263" daemon prio=10 tid=0x00007f24e831f800 nid=0x4f60 in Object.wait() [0x00007f24ed55c000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at org.apache.qpid.amqp_1_0.framing.ConnectionHandler$FrameOutput.send(ConnectionHandler.java:140)
> 	- locked <0x000000078a160c58> (a org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint)
> 	at org.apache.qpid.amqp_1_0.framing.ConnectionHandler$FrameOutput.send(ConnectionHandler.java:120)
> 	at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.send(ConnectionEndpoint.java:663)
> 	- locked <0x000000078a160c58> (a org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint)
> 	at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.send(ConnectionEndpoint.java:626)
> 	- locked <0x000000078a160c58> (a org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint)
> 	at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.sendEnd(ConnectionEndpoint.java:570)
> 	- locked <0x000000078a160c58> (a org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint)
> 	at org.apache.qpid.amqp_1_0.transport.SessionEndpoint.end(SessionEndpoint.java:176)
> 	- locked <0x000000078a160c58> (a org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint)
> 	at org.apache.qpid.amqp_1_0.transport.SessionEndpoint.end(SessionEndpoint.java:156)
> 	at org.apache.qpid.amqp_1_0.client.Session.close(Session.java:315)
> 	- locked <0x000000078cc53538> (a org.apache.qpid.amqp_1_0.client.Session)
> 	at org.apache.qpid.amqp_1_0.jms.impl.SessionImpl.close(SessionImpl.java:296)
> 	at org.apache.qpid.amqp_1_0.jms.impl.SessionImpl$1.remoteEnd(SessionImpl.java:117)
> 	at org.apache.qpid.amqp_1_0.transport.SessionEndpoint.end(SessionEndpoint.java:174)
> 	- locked <0x000000078a160c58> (a org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint)
> 	at org.apache.qpid.amqp_1_0.transport.SessionEndpoint.end(SessionEndpoint.java:156)
> 	at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.inputClosed(ConnectionEndpoint.java:428)
> 	- locked <0x000000078a160c58> (a org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint)
> 	at org.apache.qpid.amqp_1_0.client.Connection.doRead(Connection.java:444)
> 	at org.apache.qpid.amqp_1_0.client.Connection.access$000(Connection.java:51)
> 	at org.apache.qpid.amqp_1_0.client.Connection$2.run(Connection.java:275)
> 	at java.lang.Thread.run(Thread.java:744)
> "ConnectionManagementAndRecoveryWorker-id: 0" prio=10 tid=0x00007f24f80eb000 nid=0x4f41
in Object.wait() [0x00007f24a24a4000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at org.apache.qpid.amqp_1_0.framing.ConnectionHandler$FrameOutput.send(ConnectionHandler.java:140)
> 	- locked <0x000000078a160c58> (a org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint)
> 	at org.apache.qpid.amqp_1_0.framing.ConnectionHandler$FrameOutput.send(ConnectionHandler.java:120)
> 	at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.send(ConnectionEndpoint.java:663)
> 	- locked <0x000000078a160c58> (a org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint)
> 	at org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint.send(ConnectionEndpoint.java:626)
> 	- locked <0x000000078a160c58> (a org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint)
> 	at org.apache.qpid.amqp_1_0.transport.SessionEndpoint.send(SessionEndpoint.java:269)
> 	at org.apache.qpid.amqp_1_0.transport.SessionEndpoint.sendDetach(SessionEndpoint.java:765)
> 	at org.apache.qpid.amqp_1_0.transport.LinkEndpoint.detach(LinkEndpoint.java:430)
> 	- locked <0x000000078a160c58> (a org.apache.qpid.amqp_1_0.transport.ConnectionEndpoint)
> 	at org.apache.qpid.amqp_1_0.transport.LinkEndpoint.close(LinkEndpoint.java:392)
> 	at org.apache.qpid.amqp_1_0.client.Sender.close(Sender.java:398)
> 	at org.apache.qpid.amqp_1_0.jms.impl.MessageProducerImpl.close(MessageProducerImpl.java:219)
> 	at org.apache.qpid.amqp_1_0.jms.impl.SessionImpl.close(SessionImpl.java:293)
> 	at org.apache.qpid.amqp_1_0.jms.impl.ConnectionImpl.close(ConnectionImpl.java:382)
> 	



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Mime
View raw message