tapestry-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Douglas Hauge (Issue Comment Edited) (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (TAP5-1650) On a cold start with a large number of incoming requests, Tapestry can deadlock inside PlasticClassLoader/PlasticClassPool
Date Tue, 04 Oct 2011 00:31:34 GMT

    [ https://issues.apache.org/jira/browse/TAP5-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119796#comment-13119796
] 

Douglas Hauge edited comment on TAP5-1650 at 10/4/11 12:30 AM:
---------------------------------------------------------------

Attached is a potential fix for the deadlock on startup (at least after applying the patch,
we are no longer able to reproduce the deadlock). The problem was that there were multiple
orders in which the 3 Tapestry classloader locks could be acquired. 
1) standard: (prop expr conduit CL) -> (component CL) -> (service CL)
2) problematic: (componet CL) -> (prop expr conduit CL)

The second happens when *ComponentInstantiatorSourceImpl.getInstantiator* is called, but while
assembling the component classes a service class needs to be loaded. The rough path for this
is
1) ComponentInstantiatorSourceimpl.getInstantiator
2) -> ComponentInstantiatorSourceImpl.createInstantiatorForClass
3) -> PlasticManager.getClassInstantiator   (component classloader manager, not that in
'proxyFactory')
4) -> PlasticClassPool.getClassInstantiator
5) -> PlasticClassLoader.loadClass (acquires component class loader lock)
6) -> ComponentInstantiatorSourceImpl.transform -> RegistryImpl.run -> ... ->
DefaultInjectionProvider.provideInjection
7) -> ComponentClassCacheImpl.forName
8) -> ComponentClassCacheImpl.lookupClassForType (gets the property expression conduit
class loader from 'PlasticProxyFactory')
9) -> PlasticInternalUtils.toClass -> Class.forName (acquires the lock on the passed
in class loader, the property expression conduit one).

The exact stacks of the deadlocked threads vary a little, but look similar to:
Bad lock order:
{code}
"TP-Processor44":
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.tapestry5.internal.plastic.PlasticInternalUtils.toClass(PlasticInternalUtils.java:306)
at org.apache.tapestry5.internal.services.ComponentClassCacheImpl.lookupClassForType(ComponentClassCacheImpl.java:78)
at org.apache.tapestry5.internal.services.ComponentClassCacheImpl.forName(ComponentClassCacheImpl.java:65)
at $ComponentClassCache_1089cca7fe04a.forName(Unknown Source)
at org.apache.tapestry5.internal.services.DefaultInjectionProvider.provideInjection(DefaultInjectionProvider.java:65)
at $InjectionProvider2_1089cca7fe169.provideInjection(Unknown Source)
at $InjectionProvider2_1089cca7fe15d.provideInjection(Unknown Source)
at org.apache.tapestry5.internal.transform.InjectWorker$2.run(InjectWorker.java:73)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:51)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:48)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.invoke(OperationTrackerImpl.java:74)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.run(OperationTrackerImpl.java:47)
at org.apache.tapestry5.ioc.internal.PerThreadOperationTracker.run(PerThreadOperationTracker.java:76)
at org.apache.tapestry5.ioc.internal.RegistryImpl.run(RegistryImpl.java:1005)
at org.apache.tapestry5.internal.transform.InjectWorker.transform(InjectWorker.java:67)
at $ComponentClassTransformWorker2_1089cca7fe165.transform(Unknown Source)
at $ComponentClassTransformWorker2_1089cca7fe157.transform(Unknown Source)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl$3.run(ComponentInstantiatorSourceImpl.java:320)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:51)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:48)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.invoke(OperationTrackerImpl.java:74)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.run(OperationTrackerImpl.java:47)
at org.apache.tapestry5.ioc.internal.PerThreadOperationTracker.run(PerThreadOperationTracker.java:76)
at org.apache.tapestry5.ioc.internal.RegistryImpl.run(RegistryImpl.java:1005)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl.transform(ComponentInstantiatorSourceImpl.java:273)
at org.apache.tapestry5.internal.plastic.PlasticClassPool.loadAndTransformClass(PlasticClassPool.java:313)
at org.apache.tapestry5.internal.plastic.PlasticClassLoader.loadClass(PlasticClassLoader.java:38)
- locked <0x00000003027edc98> (a org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.apache.tapestry5.internal.plastic.PlasticClassPool.getClassInstantiator(PlasticClassPool.java:416)
- locked <0x00000003027edc98> (a org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at org.apache.tapestry5.plastic.PlasticManager.getClassInstantiator(PlasticManager.java:189)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl$2.invoke(ComponentInstantiatorSourceImpl.java:228)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl$2.invoke(ComponentInstantiatorSourceImpl.java:222)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.invoke(OperationTrackerImpl.java:74)
at org.apache.tapestry5.ioc.internal.PerThreadOperationTracker.invoke(PerThreadOperationTracker.java:87)
at org.apache.tapestry5.ioc.internal.RegistryImpl.invoke(RegistryImpl.java:1010)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl.createInstantiatorForClass(ComponentInstantiatorSourceImpl.java:220)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl.getInstantiator(ComponentInstantiatorSourceImpl.java:210)
- locked <0x0000000304c07bc8> (a org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl)
at $ComponentInstantiatorSource_1089cca7fe152.getInstantiator(Unknown Source)
{code}


Good lock order:
{code}
"TP-Processor100":
at org.apache.tapestry5.internal.plastic.PlasticClassLoader.loadClass(PlasticClassLoader.java:31)
- waiting to lock <0x00000003027edc98> (a org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:295)
- locked <0x000000039f889ab8> (a org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at org.apache.tapestry5.internal.plastic.PlasticClassLoader.loadClass(PlasticClassLoader.java:46)
- locked <0x000000039f889ab8> (a org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at $InternalPropertyConduit_1089cca7ffb3f.getRoot(Unknown Source)
at $InternalPropertyConduit_1089cca7ffb3f.get(Unknown Source)
at org.apache.tapestry5.internal.bindings.PropBinding.get(PropBinding.java:59)
at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.readFromBinding(ParameterWorker.java:254)
at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.get(ParameterWorker.java:372)
at lithium.tapestry.mixins.IsVisible.getfieldvalue_isVisible(IsVisible.java)
at lithium.tapestry.mixins.IsVisible.setupRender(IsVisible.java:32)
at lithium.tapestry.mixins.IsVisible.setupRender(IsVisible.java)
at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$SetupRenderPhase.invokeComponent(ComponentPageElementImpl.java:174)
at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$AbstractPhase.invoke(ComponentPageElementImpl.java:143)
at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$SetupRenderPhase.render(ComponentPageElementImpl.java:181)
at org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72)
at org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124)
{code}

                
      was (Author: dhauge):
    Attached is a potential fix for the deadlock on startup (at least after applying the patch,
we are no longer able to reproduce the deadlock). The problem was that there were multiple
orders in which the 3 Tapestry classloader locks could be acquired. 
# standard: (prop expr conduit CL) -> (component CL) -> (service CL)
# problematic: (componet CL) -> (prop expr conduit CL)

The second happens when *ComponentInstantiatorSourceImpl.getInstantiator* is called, but while
assembling the component classes a service class needs to be loaded. The rough path for this
is
# ComponentInstantiatorSourceimpl.getInstantiator
# -> ComponentInstantiatorSourceImpl.createInstantiatorForClass
# -> PlasticManager.getClassInstantiator   (component classloader manager, not that in
'proxyFactory')
# -> PlasticClassPool.getClassInstantiator
# -> PlasticClassLoader.loadClass (acquires component class loader lock)
# -> ComponentInstantiatorSourceImpl.transform -> RegistryImpl.run -> ... -> DefaultInjectionProvider.provideInjection
# -> ComponentClassCacheImpl.forName
# -> ComponentClassCacheImpl.lookupClassForType (gets the property expression conduit class
loader from 'PlasticProxyFactory')
# -> PlasticInternalUtils.toClass -> Class.forName (acquires the lock on the passed
in class loader, the property expression conduit one).

The exact stacks of the deadlocked threads vary a little, but look similar to:
Bad lock order:
{code}
"TP-Processor44":
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.tapestry5.internal.plastic.PlasticInternalUtils.toClass(PlasticInternalUtils.java:306)
at org.apache.tapestry5.internal.services.ComponentClassCacheImpl.lookupClassForType(ComponentClassCacheImpl.java:78)
at org.apache.tapestry5.internal.services.ComponentClassCacheImpl.forName(ComponentClassCacheImpl.java:65)
at $ComponentClassCache_1089cca7fe04a.forName(Unknown Source)
at org.apache.tapestry5.internal.services.DefaultInjectionProvider.provideInjection(DefaultInjectionProvider.java:65)
at $InjectionProvider2_1089cca7fe169.provideInjection(Unknown Source)
at $InjectionProvider2_1089cca7fe15d.provideInjection(Unknown Source)
at org.apache.tapestry5.internal.transform.InjectWorker$2.run(InjectWorker.java:73)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:51)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:48)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.invoke(OperationTrackerImpl.java:74)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.run(OperationTrackerImpl.java:47)
at org.apache.tapestry5.ioc.internal.PerThreadOperationTracker.run(PerThreadOperationTracker.java:76)
at org.apache.tapestry5.ioc.internal.RegistryImpl.run(RegistryImpl.java:1005)
at org.apache.tapestry5.internal.transform.InjectWorker.transform(InjectWorker.java:67)
at $ComponentClassTransformWorker2_1089cca7fe165.transform(Unknown Source)
at $ComponentClassTransformWorker2_1089cca7fe157.transform(Unknown Source)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl$3.run(ComponentInstantiatorSourceImpl.java:320)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:51)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl$1.invoke(OperationTrackerImpl.java:48)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.invoke(OperationTrackerImpl.java:74)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.run(OperationTrackerImpl.java:47)
at org.apache.tapestry5.ioc.internal.PerThreadOperationTracker.run(PerThreadOperationTracker.java:76)
at org.apache.tapestry5.ioc.internal.RegistryImpl.run(RegistryImpl.java:1005)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl.transform(ComponentInstantiatorSourceImpl.java:273)
at org.apache.tapestry5.internal.plastic.PlasticClassPool.loadAndTransformClass(PlasticClassPool.java:313)
at org.apache.tapestry5.internal.plastic.PlasticClassLoader.loadClass(PlasticClassLoader.java:38)
- locked <0x00000003027edc98> (a org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.apache.tapestry5.internal.plastic.PlasticClassPool.getClassInstantiator(PlasticClassPool.java:416)
- locked <0x00000003027edc98> (a org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at org.apache.tapestry5.plastic.PlasticManager.getClassInstantiator(PlasticManager.java:189)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl$2.invoke(ComponentInstantiatorSourceImpl.java:228)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl$2.invoke(ComponentInstantiatorSourceImpl.java:222)
at org.apache.tapestry5.ioc.internal.OperationTrackerImpl.invoke(OperationTrackerImpl.java:74)
at org.apache.tapestry5.ioc.internal.PerThreadOperationTracker.invoke(PerThreadOperationTracker.java:87)
at org.apache.tapestry5.ioc.internal.RegistryImpl.invoke(RegistryImpl.java:1010)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl.createInstantiatorForClass(ComponentInstantiatorSourceImpl.java:220)
at org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl.getInstantiator(ComponentInstantiatorSourceImpl.java:210)
- locked <0x0000000304c07bc8> (a org.apache.tapestry5.internal.services.ComponentInstantiatorSourceImpl)
at $ComponentInstantiatorSource_1089cca7fe152.getInstantiator(Unknown Source)
{code}
Good lock order:
{code}
"TP-Processor100":
at org.apache.tapestry5.internal.plastic.PlasticClassLoader.loadClass(PlasticClassLoader.java:31)
- waiting to lock <0x00000003027edc98> (a org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:295)
- locked <0x000000039f889ab8> (a org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at org.apache.tapestry5.internal.plastic.PlasticClassLoader.loadClass(PlasticClassLoader.java:46)
- locked <0x000000039f889ab8> (a org.apache.tapestry5.internal.plastic.PlasticClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at $InternalPropertyConduit_1089cca7ffb3f.getRoot(Unknown Source)
at $InternalPropertyConduit_1089cca7ffb3f.get(Unknown Source)
at org.apache.tapestry5.internal.bindings.PropBinding.get(PropBinding.java:59)
at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.readFromBinding(ParameterWorker.java:254)
at org.apache.tapestry5.internal.transform.ParameterWorker$3$1.get(ParameterWorker.java:372)
at lithium.tapestry.mixins.IsVisible.getfieldvalue_isVisible(IsVisible.java)
at lithium.tapestry.mixins.IsVisible.setupRender(IsVisible.java:32)
at lithium.tapestry.mixins.IsVisible.setupRender(IsVisible.java)
at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$SetupRenderPhase.invokeComponent(ComponentPageElementImpl.java:174)
at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$AbstractPhase.invoke(ComponentPageElementImpl.java:143)
at org.apache.tapestry5.internal.structure.ComponentPageElementImpl$SetupRenderPhase.render(ComponentPageElementImpl.java:181)
at org.apache.tapestry5.internal.services.RenderQueueImpl.run(RenderQueueImpl.java:72)
at org.apache.tapestry5.internal.services.PageRenderQueueImpl.render(PageRenderQueueImpl.java:124)
{code}

                  
> On a cold start with a large number of incoming requests, Tapestry can deadlock inside
PlasticClassLoader/PlasticClassPool
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: TAP5-1650
>                 URL: https://issues.apache.org/jira/browse/TAP5-1650
>             Project: Tapestry 5
>          Issue Type: Bug
>          Components: plastic
>    Affects Versions: 5.3
>            Reporter: Howard M. Lewis Ship
>            Assignee: Howard M. Lewis Ship
>            Priority: Critical
>              Labels: deadlock, performance
>         Attachments: tapestry-deadlock-fix-1.patch
>
>
> Discovered during load testing; the scenario is a cold start for the application then
hitting it with large numbers of concurrent requests with no ramp up. The deadlocks involve
two methods:
>      PlasticClassLoader.loadClass()
>      PlasticClassPool.getClassInstantiator()
> A proper solution would to make these methods share a single explicit Lock, rather than
be synchronized.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message