cayenne-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Huss (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CAY-1782) Deadlock when performing many concurrent inserts
Date Sun, 23 Dec 2012 04:42:13 GMT

    [ https://issues.apache.org/jira/browse/CAY-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538970#comment-13538970
] 

John Huss commented on CAY-1782:
--------------------------------

Here is an accurate account of the original issue after having stepped through it more.

Thread 1 does commitChanges (2 objects will be inserted)
Thread 1 checks out a DB connection to generate the PK for the first object
Thread 1 gets the PK and exits the pkCache lock
Thread 2 does commitChanges (2 objects will be inserted)
Thread 2 enters the pkCache lock (which is open now)
Thread 2 waits (20 seconds) for a DB connection to generate a PK (Thread 1 still has the connection).
Thread 1 tries to generate a PK for it's second object. It already has a DB connection, but
it can not enter the pkCache lock while Thread 2 holds it
Deadlock (for 20 seconds)

Here are the stack traces:

Thread t@43523: (state = BLOCKED)
**** This thread is waiting for the pkCache lock 
 - org.apache.cayenne.dba.JdbcPkGenerator.generatePk(org.apache.cayenne.access.DataNode, org.apache.cayenne.map.DbAttribute)
@bci=110, line=257 (Interpreted frame)
 - org.apache.cayenne.access.DataDomainInsertBucket.createPermIds(org.apache.cayenne.access.DbEntityClassDescriptor,
java.util.Collection) @bci=313, line=171 (Interpreted frame)
**** This thread holds a DB connection that it obtained while generating a PK for the first
object it was inserted, now it is trying to handle the second object
 - org.apache.cayenne.access.DataDomainInsertBucket.appendQueriesInternal(java.util.Collection)
@bci=161, line=76 (Interpreted frame)
 - org.apache.cayenne.access.DataDomainSyncBucket.appendQueries(java.util.Collection) @bci=18,
line=78 (Interpreted frame)
 - org.apache.cayenne.access.DataDomainFlushAction.preprocess(org.apache.cayenne.access.DataContext,
org.apache.cayenne.graph.GraphDiff) @bci=173, line=188 (Interpreted frame)
 - org.apache.cayenne.access.DataDomainFlushAction.flush(org.apache.cayenne.access.DataContext,
org.apache.cayenne.graph.GraphDiff) @bci=138, line=144 (Interpreted frame)
 - org.apache.cayenne.access.DataDomain.onSyncFlush(org.apache.cayenne.ObjectContext, org.apache.cayenne.graph.GraphDiff)
@bci=59, line=710 (Interpreted frame)
 - org.apache.cayenne.access.DataDomain$2.transform(java.lang.Object) @bci=12, line=674 (Interpreted
frame)
 - org.apache.cayenne.access.DataDomain.runInTransaction(org.apache.commons.collections.Transformer)
@bci=25, line=734 (Interpreted frame)
 - org.apache.cayenne.access.DataDomain.onSyncNoFilters(org.apache.cayenne.ObjectContext,
org.apache.cayenne.graph.GraphDiff, int) @bci=73, line=671 (Interpreted frame)
 - org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(org.apache.cayenne.ObjectContext,
org.apache.cayenne.graph.GraphDiff, int) @bci=32, line=873 (Interpreted frame)
 - org.apache.cayenne.access.DataDomain.onSync(org.apache.cayenne.ObjectContext, org.apache.cayenne.graph.GraphDiff,
int) @bci=15, line=642 (Interpreted frame)
 - org.apache.cayenne.access.DataContext.flushToParent(boolean) @bci=93, line=758 (Interpreted
frame)
 - org.apache.cayenne.access.DataContext.commitChanges() @bci=2, line=697 (Interpreted frame)
 - org.apache.cayenne.dba.ConcurrentPkGeneratorTest$1.run() @bci=58, line=68 (Interpreted
frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=441 (Interpreted frame)
 - java.util.concurrent.FutureTask$Sync.innerRun() @bci=30, line=303 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=4, line=138 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886
(Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=680 (Interpreted frame)


Thread t@43779: (state = BLOCKED)
**** This thread is waiting for a DB connection
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - org.apache.cayenne.conn.PoolManager.uncheckPooledConnection(java.lang.String, java.lang.String)
@bci=38, line=433 (Interpreted frame)
 - org.apache.cayenne.conn.PoolManager.getConnection(java.lang.String, java.lang.String) @bci=21,
line=372 (Interpreted frame)
 - org.apache.cayenne.conn.PoolManager.getConnection() @bci=9, line=361 (Interpreted frame)
 - org.apache.cayenne.access.DataNode$TransactionDataSource.getConnection() @bci=83, line=348
(Interpreted frame)
 - org.apache.cayenne.access.DataNode.performQueries(java.util.Collection, org.apache.cayenne.access.OperationObserver)
@bci=71, line=260 (Interpreted frame)
 - org.apache.cayenne.dba.JdbcPkGenerator.longPkFromDatabase(org.apache.cayenne.access.DataNode,
org.apache.cayenne.map.DbEntity) @bci=96, line=308 (Interpreted frame)
 - org.apache.cayenne.dba.JdbcPkGenerator.generatePk(org.apache.cayenne.access.DataNode, org.apache.cayenne.map.DbAttribute)
@bci=171, line=266 (Interpreted frame)
**** This thread holds the pkCache lock 
 - org.apache.cayenne.access.DataDomainInsertBucket.createPermIds(org.apache.cayenne.access.DbEntityClassDescriptor,
java.util.Collection) @bci=313, line=171 (Interpreted frame)
 - org.apache.cayenne.access.DataDomainInsertBucket.appendQueriesInternal(java.util.Collection)
@bci=161, line=76 (Interpreted frame)
 - org.apache.cayenne.access.DataDomainSyncBucket.appendQueries(java.util.Collection) @bci=18,
line=78 (Interpreted frame)
 - org.apache.cayenne.access.DataDomainFlushAction.preprocess(org.apache.cayenne.access.DataContext,
org.apache.cayenne.graph.GraphDiff) @bci=173, line=188 (Interpreted frame)
 - org.apache.cayenne.access.DataDomainFlushAction.flush(org.apache.cayenne.access.DataContext,
org.apache.cayenne.graph.GraphDiff) @bci=138, line=144 (Interpreted frame)
 - org.apache.cayenne.access.DataDomain.onSyncFlush(org.apache.cayenne.ObjectContext, org.apache.cayenne.graph.GraphDiff)
@bci=59, line=710 (Interpreted frame)
 - org.apache.cayenne.access.DataDomain$2.transform(java.lang.Object) @bci=12, line=674 (Interpreted
frame)
 - org.apache.cayenne.access.DataDomain.runInTransaction(org.apache.commons.collections.Transformer)
@bci=25, line=734 (Interpreted frame)
 - org.apache.cayenne.access.DataDomain.onSyncNoFilters(org.apache.cayenne.ObjectContext,
org.apache.cayenne.graph.GraphDiff, int) @bci=73, line=671 (Interpreted frame)
 - org.apache.cayenne.access.DataDomain$DataDomainSyncFilterChain.onSync(org.apache.cayenne.ObjectContext,
org.apache.cayenne.graph.GraphDiff, int) @bci=32, line=873 (Interpreted frame)
 - org.apache.cayenne.access.DataDomain.onSync(org.apache.cayenne.ObjectContext, org.apache.cayenne.graph.GraphDiff,
int) @bci=15, line=642 (Interpreted frame)
 - org.apache.cayenne.access.DataContext.flushToParent(boolean) @bci=93, line=758 (Interpreted
frame)
 - org.apache.cayenne.access.DataContext.commitChanges() @bci=2, line=697 (Interpreted frame)
 - org.apache.cayenne.dba.ConcurrentPkGeneratorTest$1.run() @bci=58, line=68 (Interpreted
frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=441 (Interpreted frame)
 - java.util.concurrent.FutureTask$Sync.innerRun() @bci=30, line=303 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=4, line=138 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886
(Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=680 (Interpreted frame)

So the "locks" really have to be obtained together - if you can obtain the pkCache lock, but
you don't have a DB connection yet, then you can cause a deadlock because you might be waiting
to reuse a connection from a thread that is waiting to lock the pkCache. You can avoid the
problem by moving the database call (longPkFromDatabase) OUT of the pkCache synchronized block
so that the two activities don't block each other.

The downside of doing that is that you may generate more primary keys faster than you need
because multiple threads may both try to refill the same exhausted range, and then unless
you change the prior implementation one of the generated ranges would just be thrown away.
So you want to allow for concurrent pk generation but avoid wasting entire ranges -- by using
a larger data structure, i.e. a COLLECTION of ranges instead of a single range. Implementation-wise
maintaining a mutable collection of mutable ranges would be hairy, so I elected to decompose
the range into it's elements which allows you to just use a simple collection (at the expense
of using more memory).

I've implemented the changes you suggested above - using a Queue instead (nice) and safe-guarding
the loop that replace the map entry.

                
> Deadlock when performing many concurrent inserts
> ------------------------------------------------
>
>                 Key: CAY-1782
>                 URL: https://issues.apache.org/jira/browse/CAY-1782
>             Project: Cayenne
>          Issue Type: Bug
>          Components: Core Library
>    Affects Versions: 3.0, 3.1 (final), 3.2M1
>            Reporter: John Huss
>            Assignee: John Huss
>             Fix For: 3.2M1
>
>
> I've encountered a deadlock issue in production in an app performing many INSERTs.  The
deadlock was between the PK generator and the PoolManager (getting a DB connection).  It is
very bad.  I added a unit test demonstrating the problem and a fix for it.
> The fix is possibly not ideal because it requires a larger data structure for holding
the cached primary keys, but it is far better than the previous behavior.
> If this fix is acceptable this should be back-ported to 3.1 as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message