Hi Andreas,

Thank you for the input. It was a mistake to send the same mail few times. and I apologize for that.

We have osgified the axis2 and axiom, and we re not using the wstx-asl-3.2.9.jar with in libs folder because we had some other issues with that jar. Due to that reason in [1] method of [2] class, XMLInputFactory.newInstance() returns default stax implementation (com.sun.xml.internal.stream.XMLInputFactoryImpl). This default stax implementations has a reference (not a direct one) to MultiParentClassLoader. So this causes the OOM issue.

I have reproduced it with following steps.
  1. Put the axis2.war distribution to tomcat7 (Start the server once to let the axis2.war extract)
  2. Remove the wstx-asl-3.2.9.jar from tomcat/webapps/axis2/WEB-INF/lib
  3. Put the attached axis2 services
  4. Send about 5000 requests to client service.


Even I put the osgified the wstx-asl-3.2.9.jar (regardless other issues we had), XMLInputFactory.newInstance() does not pick the [3] by reading the META-INF/services (spi)

We embedded the wstx-asl-3.2.9.jar into axiom osgified bundle, but it seems it even does not resolve the issue.

How can we make wstx-asl-3.2.9.jar only visible to axiom and let the stax implementation be [3] witch is available in wstx-asl-3.2.9.jar ?


[1] private static XMLInputFactory newXMLInputFactory(final ClassLoader classLoader,
            final StAXParserConfiguration configuration)

[2] org.apache.axiom.om.util.StAXUtils

[3] com.ctc.wstx.stax.WstxInputFactory

Thanks
Geeth

On Fri, Sep 19, 2014 at 3:14 AM, Andreas Veithen <andreas.veithen@gmail.com> wrote:
On a side note: sending the same message to the same mailing list multiple times in a short time interval generally doesn't increase the likelihood of getting a response.

Andreas


On Thu Sep 18 2014 at 4:17:39 PM Geeth Munasinghe <geethmks@gmail.com> wrote:
Hi

If anyone knows the answer for this question, please help.

Thanks
Geeth

On Wed, Sep 17, 2014 at 9:55 PM, Geeth Munasinghe <geethmks@gmail.com> wrote:
Hi all,

We are using axis2-1.6.1 version, it uses axiom 1.2.11 version, We recently encountered a OOM problem with two services which uses ServiceTCCL parameter. We are using this parameter because both those services use spring and hibernate with them. Scenario is one service is calling the other service. So one service acts as the client.

We analyzed the heap dumps and found out that OOM issue was caused by org.apache.axiom.om.util.
StAXUtils class. There are few weakhashmaps used in that class. Two of them are
  1. inputFactoryPerCLMap
  2. outputFactoryPerCLMap

And those two weak hashmaps has other weak hashmaps inside of them. Those inner hashmaps cause the issue.

We found out that when we use ServiceTCCL parameter  in the service.xml, in AbstractMessageReceiver.java  [1] of axis2 (line number 152 - 170) creates a new class loader object of the MultiParentClassLoader [2] for every request. So in StAXUtils class [3] in axiom, methods [4] and [5] uses inner  weak hashmaps of inputFactoryPerCLMap,  outputFactoryPerCLMap and fill them with the class loader as the key and XMLInputFactory / XMLOutputFactory as the value. 

Because every request gets a new class loader object of the MultiParentClassLoader, their hash values are different. So they are keep getting filled into those inner weak hash maps. Because the same key (classloder instance) gets inserted into both weak hashmaps, garbage collector does not remove them. So server goes OOM, When we analyze the heap dumps we found out that java.util.WeakHashMap fills over 80% of the memory when it goes OOM.

I have made fix in StAXUtils in axiom as follows.

Instead map.get(cl) I change it to map.get(cl.getClass().getName())

Instead map.put(cl, factory) I changed it to map.put(cl.getClass().getName(), factory)

I have attached the fix (svn diff) here with email. I am not sure I have done the correct fix for the issue. But I found that it solves my problem. Can some one please verify weather I have done the correct fix.

Please consider that upgrading to new axis2 is not a solution for us at the moment.

[1] http://svn.apache.org/repos/asf/axis/axis2/java/core/tags/v1.6.1/modules/kernel/src/org/apache/axis2/receivers/AbstractMessageReceiver.java

[2] org.apache.axis2.classloader.MultiParentClassLoader

[3] org.apache.axiom.om.util.StAXUtils

[4] getXMLInputFactory_perClassLoader(StAXParserConfiguration configuration)

[5] getXMLOutputFactory_perClassLoader(StAXWriterConfiguration configuration)


Thanks in advance.

Geeth