ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Artem Shutak <ashu...@gridgain.com>
Subject Re: Cleaning internal thread locals
Date Wed, 06 Apr 2016 08:47:53 GMT
Alexei,

I would follow the Valentin's suggestion in the issue description and would
add to Ignite tests a check on thread locals after stopping all grids. And
then we will be able to fix thread local cleaning in all places.

Thanks,
-- Artem --

On Wed, Apr 6, 2016 at 11:18 AM, Alexei Scherbakov <
alexey.scherbakoff@gmail.com> wrote:

> Valentin,
>
> I did test on demo web app where HttpSession is backed by Ignite cache and
> collect some logs.
> There are many places in Ignite where thread locals are used.
> Then cache methods are accessed in threads from servlet container pool, we
> get leaks.
> I don't think it will be easy to add thread locals cleaning in all these
> places without major code rewrite, but, for sure, it would be ideal
> solution.
>
>
> <E0><EF><F0> 05, 2016 4:21:52 PM
> org.apache.catalina.loader.WebappClassLoaderBase
> checkThreadLocalMapForLeaks
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@5fecc35f]) and a
> value of type [java.lang.Integer] (value [0]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> <E0><EF><F0> 05, 2016 4:21:52 PM
> org.apache.catalina.loader.WebappClassLoaderBase
> checkThreadLocalMapForLeaks
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1] (value
> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1@4ecdfc04])
> and a value of type [java.util.LinkedList] (value
> [[org.apache.ignite.internal.util.tostring.GridToStringThreadLocal@684a2791
> ]])
> but failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
> <E0><EF><F0> 05, 2016 4:21:52 PM
> org.apache.catalina.loader.WebappClassLoaderBase
> checkThreadLocalMapForLeaks
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1] (value
> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1@4e9300e7])
> and a value of type
> [org.apache.ignite.internal.binary.BinaryThreadLocalContext] (value
> [org.apache.ignite.internal.binary.BinaryThreadLocalContext@7934a88]) but
> failed to remove it when the web application was stopped. Threads are going
> to be renewed over time to try and avoid a probable memory leak.
> <E0><EF><F0> 05, 2016 4:21:52 PM
> org.apache.catalina.loader.WebappClassLoaderBase
> checkThreadLocalMapForLeaks
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@56502b72]) and a
> value of type
> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk]
> (value
>
> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk@4eab9d2f
> ])
> but failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
> <E0><EF><F0> 05, 2016 4:21:52 PM
> org.apache.catalina.loader.WebappClassLoaderBase
> checkThreadLocalMapForLeaks
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@eb53c9d]) and a
> value
> of type
>
> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry.StreamHolder]
> (value
>
> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry$StreamHolder@5e2c15f7
> ])
> but failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
> <E0><EF><F0> 05, 2016 4:21:52 PM
> org.apache.catalina.loader.WebappClassLoaderBase
> checkThreadLocalMapForLeaks
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@55b9da28]) and a
> value of type [java.lang.Integer] (value [0]) but failed to remove it when
> the web application was stopped. Threads are going to be renewed over time
> to try and avoid a probable memory leak.
> <E0><EF><F0> 05, 2016 4:21:52 PM
> org.apache.catalina.loader.WebappClassLoaderBase
> checkThreadLocalMapForLeaks
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1] (value
> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1@1ac73431
> ])
> and a value of type [java.lang.Integer] (value [9]) but failed to remove it
> when the web application was stopped. Threads are going to be renewed over
> time to try and avoid a probable memory leak.
> SEVERE: The web application [/app1] created a ThreadLocal with key of type
> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5] (value
> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5@28a1bd99])
> and a value of type
> [org.apache.ignite.internal.processors.cache.GridCacheAdapter.FutureHolder]
> (value
>
> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$FutureHolder@6763d70c
> ])
> but failed to remove it when the web application was stopped. Threads are
> going to be renewed over time to try and avoid a probable memory leak.
>
> As a solution, I propose to implement servlet filter as a part of Ignite
> web integration, which will be responsible for cleaning up all thread
> locals after request processing.
> Filter must be executed last in filter chain to work correctly.
> It is not necessary to use the filter in case of Tomcat, because it
> provides own solution to the problem requiring special configuration
> parameter for context
> User should decide which approach is better to him.
>
>
> 2016-04-06 1:45 GMT+03:00 Valentin Kulichenko <
> valentin.kulichenko@gmail.com
> >:
>
> > Alexei,
> >
> > I would start with defining which thread locals cause this and why they
> are
> > not properly cleaned on node shutdown. Ideally, this should never happen.
> >
> > -Val
> >
> > On Tue, Apr 5, 2016 at 10:36 AM, Alexei Scherbakov <
> > alexey.scherbakoff@gmail.com> wrote:
> >
> > > Hello.
> > >
> > > Currently I'm working on the IGNITE-967
> > > <https://issues.apache.org/jira/browse/IGNITE-967>.
> > >
> > > In fact, it is well known, easily reproduceable, ClassLoader memory
> leak
> > > problem in managed environments, such as servlet containers.
> > > See http://wiki.apache.org/tomcat/MemoryLeakProtection to read more on
> > > that.
> > > In short, the code using thread locals is responsible for cleaning them
> > > after request processing
> > > to prevent memory leaks related to different lifetime of application
> and
> > > thread in pool.
> > > Also blindly reusing thread from pool can be dangerous, because current
> > > ThreadLocal value can be different from expected default value.
> > >
> > > One posiible way of solving the problem is using post-processing
> Filter,
> > > which is responsible for cleaning all thread locals on current thread.
> > >
> > > Another approach would be sticking with container's specific solution
> to
> > > this problem.
> > > On example, tomcat provides the configuration property
> > > renewThreadsWhenStoppingContext,
> > > which forces it to renew threads involved in servicing requests for
> > > destroyed context, thus removing leaks.
> > >
> > > What do you think?
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> > >
> >
>
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message