qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sridharan, Venkatesan" <Venkatesan.Sridha...@Squarepoint-Capital.com>
Subject RE: Crash on inability to catch the internal exception thrown from qpid c++ API
Date Mon, 02 Nov 2015 00:44:58 GMT
Thanks! The original issue is that the session is closed by one thread, while it is being accessed
by another. 
Thread 1: session.nextReceiver()
Thread 2: session.close()

If this is not a supported use case of qpid API, then the bug is in my code. I had a cursory
look at the code and saw that SessionImpl had addRef()/release() and assumed the object will
not be released when in use. 

But the session was closed (even when nextReceiver() was blocking in another thread). At some
point when nextReceiver() continued execution, 
1) an exception was thrown
2)  ScopedUnlock tried to lock mutex again
3) the mutex was already gone, which threw an exception 
4) this crashed the entire program.

Stack trace on Thread 1:
#0  0x0000003d890328e5 in raise () from /lib64/libc.so.6
#1  0x0000003d890340c5 in abort () from /lib64/libc.so.6
#2  0x00007f7a429cd69b in __gnu_cxx::__verbose_terminate_handler() () from /home/replicate/dev/sentrylib/libsentry.so
#3  0x00007f7a4298cd37 in __cxxabiv1::__terminate(void (*)()) () from /home/replicate/dev/sentrylib/libsentry.so
#4  0x00007f7a4298ae5a in __cxa_call_terminate () from /home/replicate/dev/sentrylib/libsentry.so
#5  0x00007f7a4298a7c5 in __gxx_personality_v0 () from /home/replicate/dev/sentrylib/libsentry.so
#6  0x00007f7a429fa363 in _Unwind_RaiseException_Phase2 (exc=exc@entry=0x7f79f4000f80, context=context@entry=0x7f79fbffe370)
at /apps/gcc-4.9.1/libgcc/unwind.inc:62
#7  0x00007f7a429faa0a in _Unwind_RaiseException (exc=0x7f79f4000f80) at /apps/gcc-4.9.1/libgcc/unwind.inc:131
#8  0x00007f7a4298bff2 in __cxa_throw () from /home/replicate/dev/sentrylib/libsentry.so
#9  0x00007f7a4266f27f in qpid::sys::Mutex::lock (this=<optimized out>) at /apps/qpid-cpp-0.34/src/qpid/sys/posix/Mutex.h:116
#10 0x00007f7a426aaadc in ~ScopedUnlock (this=<synthetic pointer>, __in_chrg=<optimized
out>) at /apps/qpid-cpp-0.34/src/qpid/sys/Mutex.h:44
#11 qpid::client::amqp0_10::IncomingMessages::getNextDestination (this=this@entry=0x1f05690,
destination=..., timeout=...)
    at /apps/qpid-cpp-0.34/src/qpid/client/amqp0_10/IncomingMessages.cpp:202
#12 0x00007f7a426b8032 in qpid::client::amqp0_10::SessionImpl::nextReceiver (this=0x1f05640,
receiver=..., timeout=...)


Cant seem to find any qpid::RefCounted::addRef call in the entire project except in a couple
of places:
/apps/qpid-cpp-0.34/src$ find . -type f|xargs grep addRef
./qpid/RefCounted.h:    void addRef() const { ++count; }
./qpid/RefCounted.h:inline void intrusive_ptr_add_ref(const RefCounted* p) { p->addRef();
}
./qpid/legacystore/TxnCtxt.cpp:        dtokp->addRef();
./qpid/legacystore/TxnCtxt.cpp:void TxnCtxt::incrDtokRef() { dtokp->addRef(); }
./qpid/legacystore/MessageStoreImpl.cpp:            dtokp->addRef();
./qpid/legacystore/MessageStoreImpl.cpp:    ddtokp->addRef();
./qpid/linearstore/TxnCtxt.cpp:        dtokp->addRef();
./qpid/linearstore/TxnCtxt.cpp:void TxnCtxt::incrDtokRef() { dtokp->addRef(); }
./qpid/linearstore/MessageStoreImpl.cpp:            dtokp->addRef();
./qpid/linearstore/MessageStoreImpl.cpp:    ddtokp->addRef();

We can probably use the addRef()/release() machinery if the use case I highlighted is indeed
supported by qpid APi

> However in your patch, you use lambdas which though very neat would not work on older
compilers.
Ok, I will convert it to use boost::function. As an aside, are there any plans to switch to
c++11 in the future?

-----Original Message-----
From: Gordon Sim [mailto:gsim@redhat.com] 
Sent: Thursday, October 29, 2015 6:29 PM
To: dev@qpid.apache.org
Subject: Re: Crash on inability to catch the internal exception thrown from qpid c++ API

On 10/29/2015 06:14 AM, Sridharan, Venkatesan wrote:
> I have created the JIRA: 
> https://issues.apache.org/jira/browse/QPID-6811
> for the issue.
>
> I have attached a patch to the JIRA, am new to qpid development process, not sure how
to go about it - can anybody provide some pointers on how to take this patch forward?
> Tested on Linux and Windows

First off, I apologise for not replying to earlier mails. I meant to try and reproduce myself,
but didn't get the time to do so.

I tend to agree with you that the ScopedUnlock is a little unsafe. 
However in your patch, you use lambdas which though very neat would not work on older compilers.

One option (not as neat, admittedly) would be to use boost::function instead (which is already
used elsewhere in the codebase).

In terms of the original issue, i.e. if the object to which the lock instance is scoped is
being deleted while still in use by active threads, I think that may require some higher level
fix.

As for process, attaching to a JIRA is fine. There is also a reviewboard instance which is
quite nice, as it allows easy discussion of specific parts of the proposals: https://reviews.apache.org

> -----Original Message-----
> From: Sridharan, Venkatesan
> Sent: Friday, October 16, 2015 4:06 PM
> To: 'dev@qpid.apache.org' <dev@qpid.apache.org>
> Subject: RE: Crash on inability to catch the internal exception thrown 
> from qpid c++ API
>
> I finally traced the issue to throwing of an exception in ScopedUnlock destructor.
> Defined as:
> ~ScopedUnlock() { mutex.lock(); }
> Calls =>
> void Mutex::lock() {
>      QPID_POSIX_ASSERT_THROW_IF(pthread_mutex_lock(&mutex));
> }
>
> which throws and terminates the application.
>
> In my case, the ScopedUnlock dtor is called from 
> [qpid-cpp-0.34/src/qpid/client/amqp0_10/IncomingMessages.cpp:202]
>
> I am using the Session object from multiple threads and I suspect the one thread calls
close() on session while another is operating on it. Is this scenario supported?
>
> Scoped unlock does not seem the correct approach to me. Instead of freeing a resource
in the destructor whose failure is not catastrophic, ScopedUnlock tries to acquire a resource
on exit which can fail and there is no way this can be communicated back to the code!
>
> I propose that the class be done away with and an explicit resource acquisition with
proper failure handling semantics be used.
>
>
> -----Original Message-----
> From: Sridharan, Venkatesan
> Sent: Monday, October 12, 2015 10:54 AM
> To: 'dev@qpid.apache.org' <dev@qpid.apache.org>
> Subject: Crash on inability to catch the internal exception thrown 
> from qpid c++ API
>
> Hi Folks,
>
> While debugging an application crash in a client program using qpid c++ API, I traced
the root cause of the error to the fact that an internal exception (qpid::Exception) which
is not exposed in the API was thrown.  The exception was in Mutex.h:116 locking code. I had
coded an error recovery  part of the code and looks like:
>
> // qpid::messaging::Connection m_conn; // qpid::messaging::Session 
> m_session try {
> 	if (m_session.hasError())
> 	{	
> 		m_session.close();
> 		m_conn.close();
> 		LOG(WARNING) <<  "Session has errors, recreating...";
> 		m_conn.open();
> 		m_session = m_conn.createSession();
> 	}
> }
> catch(std::exception const& ex)
> {
> ..
> }
> catch(...)
> {
> ..
> }
>
> Exception is not caught in both the catch() blocks.
>
> This is because there are places in the control flow where a Mutex object is locked but
the internal exception thrown is not captured. This leads to leak of internal exception and
since it is not exposed in the API, gcc cannot get its typeinfo for exception handling and
terminate() gets called.  Since the issue is with the QPid API, I am powerless to prevent
crashes in my application.
>
> Using two exception hierarchies and  leaking internal exceptions is bug prone and I think
we should expose all the exception types (including internal) in the API so as to not crash
the clients using the API.  What is the consensus here on exposing ALL the exceptions qpid
client API might possibly throw? Or am I missing something?
>
> Cheers
> Venkatesan
>
>
>
>
> Confidentiality Note:  This e-mail and any attachments are 
> confidential and may be protected by legal privilege. If you are not 
> the intended recipient, be aware that any disclosure, copying, 
> distribution or use of this e-mail or any attachment is prohibited. If 
> you have received this e-mail in error, please notify us immediately 
> by returning it to the sender and delete this copy from your system. Thank you for your
cooperation.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org For additional 
> commands, e-mail: dev-help@qpid.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org For additional commands, e-mail: dev-help@qpid.apache.org




Confidentiality Note:  This e-mail and any attachments are confidential and may
be protected by legal privilege. If you are not the intended recipient, be aware
that any disclosure, copying, distribution or use of this e-mail or any
attachment is prohibited. If you have received this e-mail in error, please
notify us immediately by returning it to the sender and delete this copy from
your system. Thank you for your cooperation.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Mime
View raw message