james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Charles <eric.char...@u-mangate.com>
Subject Re: OutOfMemory exception
Date Thu, 01 Apr 2010 07:16:36 GMT
Hi Norman,

Tks for the smpt-source pointer.
I looked at it, but finally, I think I will develop those little classes 
to have more control on it.
I will put it on github.

I launched latest trunk james yesterday evening.
During the night, 3 hours after the launch, the following occured:

KahDB slower and slower...

INFO  23:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase | 
Slow KahaDB access: cleanup took 2330
INFO  23:38:04,591 | james.mailetcontext | Error while storing mail.
org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be 
parsed.;
   nested exception is:
     org.apache.james.imap.mailbox.StorageException: failed. Transaction 
commit failed.;
   nested exception is:
<openjpa-1.2.1-r752877:753278 fatal store error> 
org.apache.openjpa.persistence.RollbackException: The transaction has 
been rolled back.  See the nested exceptions for details on the errors 
that occurred.
     at 
org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265)

     at 
org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:130)
     ... 55 more
Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java 
exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt 
1363215207 INSERT INTO Message (id, bodyStartOctet, content, 
contentOctets, mediaType, subType, textu
alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=(long) 2678, (int) 
1260, (byte[]) [B@7a00e70c, (long) 47887, (String) text, (String) html, 
(long) 876]} [code=0, state=XJ001]
     at 
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192)
     at 
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57)
     at 
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866)
     at 
org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269)

So still a OOM exception that was shown by yet-another-component (in 
this case, the StoreMailbox).


There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its 
job).
All 4 files were 0 bytes.

I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB 
max memory).

With the previous parameters ( -Xmx512m), the process was taking the 
whole 512MB.

Tks,
Eric


On 03/31/2010 08:26 PM, Norman Maurer wrote:
> Hi eric,
>
> thx for all your help, maybe smtp-source ( which is included in
> postfix ) can help you with stress testing.
>
> Bye
> Norman
>
> 2010/3/31, Eric Charles<eric.charles@u-mangate.com>:
>    
>> Hi Norman,
>>
>> There are not so much .m64 files, so even it it shows that a mail
>> processing didn't reach the end and may have cause a leak, I don't think
>> it would make crash my process so quick (between 10 and 24 hours).
>>
>> I quickly analysed a few dumps last week : there is always a class that
>> took 30/40% of the memory. Sometimes a activemq class, sometimes not
>> (don't remember the details).
>> Also notable, the stack trace (the place where the OOM gives problems)
>> varies.
>> Today stack was:
>> org.apache.camel.RuntimeCamelException:
>> org.apache.camel.CamelExchangeException: Error processing Exchange.
>> Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102,
>> responseRequired = true, messageId =
>> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination = null,
>> originalTransactionId = null, producerId =
>> ID:srv001-51032-1270011735798-2:0:24:1, destination =
>> queue://processor.root, transactionId = null, expiration = 0, timestamp
>> = 1270014325685, arrival = 0, brokerInTime = 1270014326893,
>> brokerOutTime = 1270014333023, correlationId = null, replyTo = null,
>> persistent = true, type = null, priority = 4, groupID = null,
>> groupSequence = 0, targetConsumerId = null, compressed = false, userID =
>> null, content = org.apache.activemq.util.ByteSequence@69b568d0,
>> marshalledProperties = null, dataStructure = null, redeliveryCounter =
>> 0, size = 10718, properties = null, readOnlyProperties = true,
>> readOnlyBody = true, droppable = false}]. Caused by:
>> [java.lang.OutOfMemoryError - Java heap space]
>>       at
>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055)
>>       at
>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154)
>> ...
>>
>>
>>
>> I also think the tmp files are not the main cause.
>>
>> I have now james running in eclipse and I made already made a little
>> trip in the code. Quite impressive since last time I looked at it.
>> I will try to stress james with a small smtp/pop3 client (I used
>> postage, but a simple class with commons-net behind) may be easier and
>> see what happens (eventually with eclipse profiler).
>>
>> It will be for this weekend for me.
>>
>> Bye,
>>
>> Eric
>>
>>
>> On 03/31/2010 07:11 PM, Norman Maurer wrote:
>>      
>>> Hi Eric,
>>>
>>> thx for keeping us in the loop... I'm still not sure why the .m64
>>> files are still in the tmp folder sometimes.. But I suspect the files
>>> are not the cause of the OOM.
>>>
>>> I didn't get a OOM since yesterday morning (the time I deployed
>>> current trunk version), but just found to new .m64 files..
>>>
>>> So I'm still searching for the real cause. If nothing helps I will
>>> need to use a profiler to find th leak.
>>>
>>> Bye,
>>> Norman
>>>
>>> 2010/3/31 Eric Charles<eric.charles@u-mangate.com>:
>>>
>>>        
>>>> Hi Norman,
>>>>
>>>> I had defined the Null mailet
>>>>
>>>> <mailet match="HostIsLocal" class="Null">
>>>> <processor>   local-address-error</processor>
>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>> </mailet>
>>>>
>>>> but now, I use the standard config
>>>>
>>>> <mailet match="HostIsLocal" class="ToProcessor">
>>>> <processor>   local-address-error</processor>
>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>> </mailet>
>>>>
>>>> I still have the OOM.
>>>>
>>>> I will now deploy a fresh svn with your last commits and enable the
>>>> ValidRcptHandler.
>>>> I keep you posted with the result,
>>>>
>>>> Tks,
>>>> Eric
>>>>
>>>>
>>>> On 03/30/2010 06:49 AM, Norman Maurer wrote:
>>>>
>>>>          
>>>>> Hi Eric,
>>>>>
>>>>> you said all the files are related to address-errors , could you show
>>>>> me your address error processor config?
>>>>> Does the Problem still exist when you enable the ValidRcptHandler in
>>>>> the smtpserver.xml file?
>>>>>
>>>>> Thx
>>>>> Norman
>>>>>
>>>>> 2010/3/30, Eric Charles<eric.charles@u-mangate.com>:
>>>>>
>>>>>
>>>>>            
>>>>>> Oops, no, the files are still there (only unknown@known.com).
>>>>>> Eric
>>>>>>
>>>>>>
>>>>>> On 03/29/2010 10:16 PM, Eric Charles wrote:
>>>>>>
>>>>>>
>>>>>>              
>>>>>>> Hi Norman,
>>>>>>>
>>>>>>> I just deployed your new commit with the new camel DisposeProcess.
>>>>>>> Good news : I don't see anymore the m64 file in tmp (well I see
1
>>>>>>> file, on the second after, it is no more there, so the dispose
works
>>>>>>> as it should).
>>>>>>>
>>>>>>> I keep you posted with our eventual future OOM.
>>>>>>>
>>>>>>> Tks,
>>>>>>> Eric
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote:
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>>>> Hi Norman,
>>>>>>>>
>>>>>>>> The .m64 are all to "unkown user" but to "well known domain"
(so for
>>>>>>>> <unknown@known.com>).
>>>>>>>> They are from various size (with and without attachment).
>>>>>>>> They are well formed (I can open the downloaded file with
>>>>>>>> thunderbird)
>>>>>>>>
>>>>>>>> However, when I sent a mail to unknown@known.com, I don't
see it in
>>>>>>>> the /tmp
>>>>>>>>
>>>>>>>> Tks,
>>>>>>>> Eric
>>>>>>>>
>>>>>>>>
>>>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>                  
>>>>>>>>> Hi Eric,
>>>>>>>>>
>>>>>>>>> sure.. we have to find the OOM cause. What would be interesting,
>>>>>>>>> could
>>>>>>>>> you check somehow if the .m64 files are files which are
related to
>>>>>>>>> successfully delivered mail ?
>>>>>>>>>
>>>>>>>>> Thx,
>>>>>>>>> Norman
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2010/3/29 Eric Charles<eric.charles@u-mangate.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                    
>>>>>>>>>> Norman,
>>>>>>>>>>
>>>>>>>>>> Done. Now, we have to wait...
>>>>>>>>>>
>>>>>>>>>> I saw some created *.m64 that were removed.
>>>>>>>>>> But there are other ones that remains in /tmp.
>>>>>>>>>>
>>>>>>>>>> I sometimes run a jmap (java memory map) to produce
a heap dump and
>>>>>>>>>> analyse
>>>>>>>>>> it.
>>>>>>>>>> At the beginning, everything seems ok, and often,
when I come back,
>>>>>>>>>> the OOM
>>>>>>>>>> has already occured.
>>>>>>>>>>
>>>>>>>>>> I changed the -Xmx512m to -Xmx256m to have a quicker
exception (if
>>>>>>>>>> any,
>>>>>>>>>> let's hope not).
>>>>>>>>>>
>>>>>>>>>> The *.m64 are a clue and we should ensure that they
are removed in
>>>>>>>>>> all
>>>>>>>>>> circumstances.
>>>>>>>>>> This may be the easy part, as we have some visible
clues (the
>>>>>>>>>> files).
>>>>>>>>>> We should also ensure after that there are no other
causes to the
>>>>>>>>>> leaks.
>>>>>>>>>>
>>>>>>>>>> Tks,
>>>>>>>>>> Eric
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                      
>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>
>>>>>>>>>>> I found the cause for the not deleted temporary
files. Hopefully
>>>>>>>>>>> this
>>>>>>>>>>> is the cause of the OOM. Could try to svn up
and run again ?
>>>>>>>>>>>
>>>>>>>>>>> Thx,
>>>>>>>>>>> Norman
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2010/3/29 Norman Maurer<norman.maurer@googlemail.com>:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                        
>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>
>>>>>>>>>>>> thx for the report. I see exact the same
problem today here..
>>>>>>>>>>>> (The
>>>>>>>>>>>> OOM). Didn't notice the files in the tmp
folder, but I think
>>>>>>>>>>>> thats
>>>>>>>>>>>> a
>>>>>>>>>>>> good pointer. I will try to debug the problem
later or tomorrow.
>>>>>>>>>>>> But I
>>>>>>>>>>>> suspect you are right about the tmp files
and jms producers.. Did
>>>>>>>>>>>> you
>>>>>>>>>>>> run a kill -3 pid to see what threads are
active etc ?
>>>>>>>>>>>>
>>>>>>>>>>>> Thx,
>>>>>>>>>>>> Norman
>>>>>>>>>>>>
>>>>>>>>>>>> Ps: Patches are welcome :)
>>>>>>>>>>>>
>>>>>>>>>>>> 2010/3/29 Eric Charles<eric.charles@u-mangate.com>:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                          
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> The last two weeks, I deployed various
trunk snapshots.
>>>>>>>>>>>>> After james running less than 1 day,
I always ran into a
>>>>>>>>>>>>> OutOfMemory
>>>>>>>>>>>>> Exception.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I analysed the logs (for example STACK
TRACE 1 in annex), and
>>>>>>>>>>>>> always
>>>>>>>>>>>>> found
>>>>>>>>>>>>> Camel complaining when trying to deliver
on JMS queue.
>>>>>>>>>>>>> I tried to adapt the ActiveMQ configuration
based on
>>>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html
but nothing
>>>>>>>>>>>>> helped.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I also analyzed various heap dump :sometimes,
65% was used by
>>>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession,
sometimes
>>>>>>>>>>>>> by a
>>>>>>>>>>>>> activemq class,...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                            
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                    
>>>>>>>>
>>>>>>>>                  
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>              
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>            
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>>          
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>        
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>      
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
>
>    


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message