james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Norman Maurer <norman.mau...@googlemail.com>
Subject Re: OutOfMemory exception
Date Thu, 01 Apr 2010 07:24:36 GMT
Hi Eric,

thx again for your tests...

So I think we have two independent problems here:

1) The .m64 files which are not deleted always:

My guess here is that there is something wrong with processing Mail
objects with state Null. I will investigate futher.. I also found a
possible problem with .m64 files which get not deleted when a
Exception is throws while processing the camel routes. I will patch
this soon.

2) The OOM

My guess here is that ActiveMQ is not freeing up memory. Not sure why yet

I think 2) is the more criticial problem and should get higher
priority to get fixed..

Bye,
Norman



2010/4/1 Eric Charles <eric.charles@u-mangate.com>:
> Hi Norman,
>
> Tks for the smpt-source pointer.
> I looked at it, but finally, I think I will develop those little classes to
> have more control on it.
> I will put it on github.
>
> I launched latest trunk james yesterday evening.
> During the night, 3 hours after the launch, the following occured:
>
> KahDB slower and slower...
>
> INFO  23:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
> KahaDB access: cleanup took 2330
> INFO  23:38:04,591 | james.mailetcontext | Error while storing mail.
> org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be
> parsed.;
>  nested exception is:
>    org.apache.james.imap.mailbox.StorageException: failed. Transaction
> commit failed.;
>  nested exception is:
> <openjpa-1.2.1-r752877:753278 fatal store error>
> org.apache.openjpa.persistence.RollbackException: The transaction has been
> rolled back.  See the nested exceptions for details on the errors that
> occurred.
>    at
> org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265)
>
>    at
> org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:130)
>    ... 55 more
> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
> 1363215207 INSERT INTO Message (id, bodyStartOctet, content, contentOctets,
> mediaType, subType, textu
> alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=(long) 2678, (int) 1260,
> (byte[]) [B@7a00e70c, (long) 47887, (String) text, (String) html, (long)
> 876]} [code=0, state=XJ001]
>    at
> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192)
>    at
> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57)
>    at
> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866)
>    at
> org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269)
>
> So still a OOM exception that was shown by yet-another-component (in this
> case, the StoreMailbox).
>
>
> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
> job).
> All 4 files were 0 bytes.
>
> I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB max
> memory).
>
> With the previous parameters ( -Xmx512m), the process was taking the whole
> 512MB.
>
> Tks,
> Eric
>
>
> On 03/31/2010 08:26 PM, Norman Maurer wrote:
>>
>> Hi eric,
>>
>> thx for all your help, maybe smtp-source ( which is included in
>> postfix ) can help you with stress testing.
>>
>> Bye
>> Norman
>>
>> 2010/3/31, Eric Charles<eric.charles@u-mangate.com>:
>>
>>>
>>> Hi Norman,
>>>
>>> There are not so much .m64 files, so even it it shows that a mail
>>> processing didn't reach the end and may have cause a leak, I don't think
>>> it would make crash my process so quick (between 10 and 24 hours).
>>>
>>> I quickly analysed a few dumps last week : there is always a class that
>>> took 30/40% of the memory. Sometimes a activemq class, sometimes not
>>> (don't remember the details).
>>> Also notable, the stack trace (the place where the OOM gives problems)
>>> varies.
>>> Today stack was:
>>> org.apache.camel.RuntimeCamelException:
>>> org.apache.camel.CamelExchangeException: Error processing Exchange.
>>> Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102,
>>> responseRequired = true, messageId =
>>> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination = null,
>>> originalTransactionId = null, producerId =
>>> ID:srv001-51032-1270011735798-2:0:24:1, destination =
>>> queue://processor.root, transactionId = null, expiration = 0, timestamp
>>> = 1270014325685, arrival = 0, brokerInTime = 1270014326893,
>>> brokerOutTime = 1270014333023, correlationId = null, replyTo = null,
>>> persistent = true, type = null, priority = 4, groupID = null,
>>> groupSequence = 0, targetConsumerId = null, compressed = false, userID =
>>> null, content = org.apache.activemq.util.ByteSequence@69b568d0,
>>> marshalledProperties = null, dataStructure = null, redeliveryCounter =
>>> 0, size = 10718, properties = null, readOnlyProperties = true,
>>> readOnlyBody = true, droppable = false}]. Caused by:
>>> [java.lang.OutOfMemoryError - Java heap space]
>>>      at
>>>
>>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055)
>>>      at
>>>
>>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154)
>>> ...
>>>
>>>
>>>
>>> I also think the tmp files are not the main cause.
>>>
>>> I have now james running in eclipse and I made already made a little
>>> trip in the code. Quite impressive since last time I looked at it.
>>> I will try to stress james with a small smtp/pop3 client (I used
>>> postage, but a simple class with commons-net behind) may be easier and
>>> see what happens (eventually with eclipse profiler).
>>>
>>> It will be for this weekend for me.
>>>
>>> Bye,
>>>
>>> Eric
>>>
>>>
>>> On 03/31/2010 07:11 PM, Norman Maurer wrote:
>>>
>>>>
>>>> Hi Eric,
>>>>
>>>> thx for keeping us in the loop... I'm still not sure why the .m64
>>>> files are still in the tmp folder sometimes.. But I suspect the files
>>>> are not the cause of the OOM.
>>>>
>>>> I didn't get a OOM since yesterday morning (the time I deployed
>>>> current trunk version), but just found to new .m64 files..
>>>>
>>>> So I'm still searching for the real cause. If nothing helps I will
>>>> need to use a profiler to find th leak.
>>>>
>>>> Bye,
>>>> Norman
>>>>
>>>> 2010/3/31 Eric Charles<eric.charles@u-mangate.com>:
>>>>
>>>>
>>>>>
>>>>> Hi Norman,
>>>>>
>>>>> I had defined the Null mailet
>>>>>
>>>>> <mailet match="HostIsLocal" class="Null">
>>>>> <processor>   local-address-error</processor>
>>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>>> </mailet>
>>>>>
>>>>> but now, I use the standard config
>>>>>
>>>>> <mailet match="HostIsLocal" class="ToProcessor">
>>>>> <processor>   local-address-error</processor>
>>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>>> </mailet>
>>>>>
>>>>> I still have the OOM.
>>>>>
>>>>> I will now deploy a fresh svn with your last commits and enable the
>>>>> ValidRcptHandler.
>>>>> I keep you posted with the result,
>>>>>
>>>>> Tks,
>>>>> Eric
>>>>>
>>>>>
>>>>> On 03/30/2010 06:49 AM, Norman Maurer wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Hi Eric,
>>>>>>
>>>>>> you said all the files are related to address-errors , could you
show
>>>>>> me your address error processor config?
>>>>>> Does the Problem still exist when you enable the ValidRcptHandler
in
>>>>>> the smtpserver.xml file?
>>>>>>
>>>>>> Thx
>>>>>> Norman
>>>>>>
>>>>>> 2010/3/30, Eric Charles<eric.charles@u-mangate.com>:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Oops, no, the files are still there (only unknown@known.com).
>>>>>>> Eric
>>>>>>>
>>>>>>>
>>>>>>> On 03/29/2010 10:16 PM, Eric Charles wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Hi Norman,
>>>>>>>>
>>>>>>>> I just deployed your new commit with the new camel DisposeProcess.
>>>>>>>> Good news : I don't see anymore the m64 file in tmp (well
I see 1
>>>>>>>> file, on the second after, it is no more there, so the dispose
works
>>>>>>>> as it should).
>>>>>>>>
>>>>>>>> I keep you posted with our eventual future OOM.
>>>>>>>>
>>>>>>>> Tks,
>>>>>>>> Eric
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Norman,
>>>>>>>>>
>>>>>>>>> The .m64 are all to "unkown user" but to "well known
domain" (so
>>>>>>>>> for
>>>>>>>>> <unknown@known.com>).
>>>>>>>>> They are from various size (with and without attachment).
>>>>>>>>> They are well formed (I can open the downloaded file
with
>>>>>>>>> thunderbird)
>>>>>>>>>
>>>>>>>>> However, when I sent a mail to unknown@known.com, I don't
see it in
>>>>>>>>> the /tmp
>>>>>>>>>
>>>>>>>>> Tks,
>>>>>>>>> Eric
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Eric,
>>>>>>>>>>
>>>>>>>>>> sure.. we have to find the OOM cause. What would
be interesting,
>>>>>>>>>> could
>>>>>>>>>> you check somehow if the .m64 files are files which
are related to
>>>>>>>>>> successfully delivered mail ?
>>>>>>>>>>
>>>>>>>>>> Thx,
>>>>>>>>>> Norman
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2010/3/29 Eric Charles<eric.charles@u-mangate.com>:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Norman,
>>>>>>>>>>>
>>>>>>>>>>> Done. Now, we have to wait...
>>>>>>>>>>>
>>>>>>>>>>> I saw some created *.m64 that were removed.
>>>>>>>>>>> But there are other ones that remains in /tmp.
>>>>>>>>>>>
>>>>>>>>>>> I sometimes run a jmap (java memory map) to produce
a heap dump
>>>>>>>>>>> and
>>>>>>>>>>> analyse
>>>>>>>>>>> it.
>>>>>>>>>>> At the beginning, everything seems ok, and often,
when I come
>>>>>>>>>>> back,
>>>>>>>>>>> the OOM
>>>>>>>>>>> has already occured.
>>>>>>>>>>>
>>>>>>>>>>> I changed the -Xmx512m to -Xmx256m to have a
quicker exception
>>>>>>>>>>> (if
>>>>>>>>>>> any,
>>>>>>>>>>> let's hope not).
>>>>>>>>>>>
>>>>>>>>>>> The *.m64 are a clue and we should ensure that
they are removed
>>>>>>>>>>> in
>>>>>>>>>>> all
>>>>>>>>>>> circumstances.
>>>>>>>>>>> This may be the easy part, as we have some visible
clues (the
>>>>>>>>>>> files).
>>>>>>>>>>> We should also ensure after that there are no
other causes to the
>>>>>>>>>>> leaks.
>>>>>>>>>>>
>>>>>>>>>>> Tks,
>>>>>>>>>>> Eric
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>
>>>>>>>>>>>> I found the cause for the not deleted temporary
files. Hopefully
>>>>>>>>>>>> this
>>>>>>>>>>>> is the cause of the OOM. Could try to svn
up and run again ?
>>>>>>>>>>>>
>>>>>>>>>>>> Thx,
>>>>>>>>>>>> Norman
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2010/3/29 Norman Maurer<norman.maurer@googlemail.com>:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>
>>>>>>>>>>>>> thx for the report. I see exact the same
problem today here..
>>>>>>>>>>>>> (The
>>>>>>>>>>>>> OOM). Didn't notice the files in the
tmp folder, but I think
>>>>>>>>>>>>> thats
>>>>>>>>>>>>> a
>>>>>>>>>>>>> good pointer. I will try to debug the
problem later or
>>>>>>>>>>>>> tomorrow.
>>>>>>>>>>>>> But I
>>>>>>>>>>>>> suspect you are right about the tmp files
and jms producers..
>>>>>>>>>>>>> Did
>>>>>>>>>>>>> you
>>>>>>>>>>>>> run a kill -3 pid to see what threads
are active etc ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ps: Patches are welcome :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2010/3/29 Eric Charles<eric.charles@u-mangate.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The last two weeks, I deployed various
trunk snapshots.
>>>>>>>>>>>>>> After james running less than 1 day,
I always ran into a
>>>>>>>>>>>>>> OutOfMemory
>>>>>>>>>>>>>> Exception.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I analysed the logs (for example
STACK TRACE 1 in annex), and
>>>>>>>>>>>>>> always
>>>>>>>>>>>>>> found
>>>>>>>>>>>>>> Camel complaining when trying to
deliver on JMS queue.
>>>>>>>>>>>>>> I tried to adapt the ActiveMQ configuration
based on
>>>>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html
but
>>>>>>>>>>>>>> nothing
>>>>>>>>>>>>>> helped.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I also analyzed various heap dump
:sometimes, 65% was used by
>>>>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession,
>>>>>>>>>>>>>> sometimes
>>>>>>>>>>>>>> by a
>>>>>>>>>>>>>> activemq class,...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message