ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@gridgain.com>
Subject Re: Cleaning internal thread locals
Date Wed, 06 Apr 2016 08:50:54 GMT
As far as I understand Alexei, the issue here is that Ignite kernal 
doesn't have list of all the threads that accessed it during its lifetime.
So the kernal won't be able to iterate over all the threads (like 
threads from Tomcat thread pool) and perform all the cleaning. This is 
the issue.

--
Denis

On 4/6/2016 11:47 AM, Artem Shutak wrote:
> Alexei,
>
> I would follow the Valentin's suggestion in the issue description and would
> add to Ignite tests a check on thread locals after stopping all grids. And
> then we will be able to fix thread local cleaning in all places.
>
> Thanks,
> -- Artem --
>
> On Wed, Apr 6, 2016 at 11:18 AM, Alexei Scherbakov <
> alexey.scherbakoff@gmail.com> wrote:
>
>> Valentin,
>>
>> I did test on demo web app where HttpSession is backed by Ignite cache and
>> collect some logs.
>> There are many places in Ignite where thread locals are used.
>> Then cache methods are accessed in threads from servlet container pool, we
>> get leaks.
>> I don't think it will be easy to add thread locals cleaning in all these
>> places without major code rewrite, but, for sure, it would be ideal
>> solution.
>>
>>
>> <E0><EF><F0> 05, 2016 4:21:52 PM
>> org.apache.catalina.loader.WebappClassLoaderBase
>> checkThreadLocalMapForLeaks
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@5fecc35f]) and a
>> value of type [java.lang.Integer] (value [0]) but failed to remove it when
>> the web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> <E0><EF><F0> 05, 2016 4:21:52 PM
>> org.apache.catalina.loader.WebappClassLoaderBase
>> checkThreadLocalMapForLeaks
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1] (value
>> [org.apache.ignite.internal.util.tostring.GridToStringBuilder$1@4ecdfc04])
>> and a value of type [java.util.LinkedList] (value
>> [[org.apache.ignite.internal.util.tostring.GridToStringThreadLocal@684a2791
>> ]])
>> but failed to remove it when the web application was stopped. Threads are
>> going to be renewed over time to try and avoid a probable memory leak.
>> <E0><EF><F0> 05, 2016 4:21:52 PM
>> org.apache.catalina.loader.WebappClassLoaderBase
>> checkThreadLocalMapForLeaks
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1] (value
>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext$1@4e9300e7])
>> and a value of type
>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext] (value
>> [org.apache.ignite.internal.binary.BinaryThreadLocalContext@7934a88]) but
>> failed to remove it when the web application was stopped. Threads are going
>> to be renewed over time to try and avoid a probable memory leak.
>> <E0><EF><F0> 05, 2016 4:21:52 PM
>> org.apache.catalina.loader.WebappClassLoaderBase
>> checkThreadLocalMapForLeaks
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@56502b72]) and a
>> value of type
>> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk]
>> (value
>>
>> [org.apache.ignite.internal.binary.streams.BinaryMemoryAllocatorChunk@4eab9d2f
>> ])
>> but failed to remove it when the web application was stopped. Threads are
>> going to be renewed over time to try and avoid a probable memory leak.
>> <E0><EF><F0> 05, 2016 4:21:52 PM
>> org.apache.catalina.loader.WebappClassLoaderBase
>> checkThreadLocalMapForLeaks
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@eb53c9d]) and a
>> value
>> of type
>>
>> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry.StreamHolder]
>> (value
>>
>> [org.apache.ignite.marshaller.optimized.OptimizedObjectStreamRegistry$StreamHolder@5e2c15f7
>> ])
>> but failed to remove it when the web application was stopped. Threads are
>> going to be renewed over time to try and avoid a probable memory leak.
>> <E0><EF><F0> 05, 2016 4:21:52 PM
>> org.apache.catalina.loader.WebappClassLoaderBase
>> checkThreadLocalMapForLeaks
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1] (value
>> [org.apache.ignite.internal.util.GridSpinReadWriteLock$1@55b9da28]) and a
>> value of type [java.lang.Integer] (value [0]) but failed to remove it when
>> the web application was stopped. Threads are going to be renewed over time
>> to try and avoid a probable memory leak.
>> <E0><EF><F0> 05, 2016 4:21:52 PM
>> org.apache.catalina.loader.WebappClassLoaderBase
>> checkThreadLocalMapForLeaks
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1] (value
>> [org.apache.ignite.internal.util.StripedCompositeReadWriteLock$1@1ac73431
>> ])
>> and a value of type [java.lang.Integer] (value [9]) but failed to remove it
>> when the web application was stopped. Threads are going to be renewed over
>> time to try and avoid a probable memory leak.
>> SEVERE: The web application [/app1] created a ThreadLocal with key of type
>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5] (value
>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$5@28a1bd99])
>> and a value of type
>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter.FutureHolder]
>> (value
>>
>> [org.apache.ignite.internal.processors.cache.GridCacheAdapter$FutureHolder@6763d70c
>> ])
>> but failed to remove it when the web application was stopped. Threads are
>> going to be renewed over time to try and avoid a probable memory leak.
>>
>> As a solution, I propose to implement servlet filter as a part of Ignite
>> web integration, which will be responsible for cleaning up all thread
>> locals after request processing.
>> Filter must be executed last in filter chain to work correctly.
>> It is not necessary to use the filter in case of Tomcat, because it
>> provides own solution to the problem requiring special configuration
>> parameter for context
>> User should decide which approach is better to him.
>>
>>
>> 2016-04-06 1:45 GMT+03:00 Valentin Kulichenko <
>> valentin.kulichenko@gmail.com
>>> :
>>> Alexei,
>>>
>>> I would start with defining which thread locals cause this and why they
>> are
>>> not properly cleaned on node shutdown. Ideally, this should never happen.
>>>
>>> -Val
>>>
>>> On Tue, Apr 5, 2016 at 10:36 AM, Alexei Scherbakov <
>>> alexey.scherbakoff@gmail.com> wrote:
>>>
>>>> Hello.
>>>>
>>>> Currently I'm working on the IGNITE-967
>>>> <https://issues.apache.org/jira/browse/IGNITE-967>.
>>>>
>>>> In fact, it is well known, easily reproduceable, ClassLoader memory
>> leak
>>>> problem in managed environments, such as servlet containers.
>>>> See http://wiki.apache.org/tomcat/MemoryLeakProtection to read more on
>>>> that.
>>>> In short, the code using thread locals is responsible for cleaning them
>>>> after request processing
>>>> to prevent memory leaks related to different lifetime of application
>> and
>>>> thread in pool.
>>>> Also blindly reusing thread from pool can be dangerous, because current
>>>> ThreadLocal value can be different from expected default value.
>>>>
>>>> One posiible way of solving the problem is using post-processing
>> Filter,
>>>> which is responsible for cleaning all thread locals on current thread.
>>>>
>>>> Another approach would be sticking with container's specific solution
>> to
>>>> this problem.
>>>> On example, tomcat provides the configuration property
>>>> renewThreadsWhenStoppingContext,
>>>> which forces it to renew threads involved in servicing requests for
>>>> destroyed context, thus removing leaks.
>>>>
>>>> What do you think?
>>>>
>>>>
>>>> --
>>>>
>>>> Best regards,
>>>> Alexei Scherbakov
>>>>
>>
>>
>> --
>>
>> Best regards,
>> Alexei Scherbakov
>>


Mime
View raw message