james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Charles <eric.char...@u-mangate.com>
Subject Re: OutOfMemory exception
Date Thu, 01 Apr 2010 07:38:17 GMT
2) Launching james, memory usage is 300MB. This seems normal and holding 
the traffic well. At a certain time, memory jumps to 3GB (I gave much 
memory via -Xmx).
Maybe a certain condition produces this.
Debugging in eclipse shows me that many threads are created by camel. I 
also had the impression that some threads were falling and other ones 
were recreated. If this is the case, is it normal.
I'm going to look at activemq/camel (I don't know them really), and also 
still read the http://activemq.apache.org/javalangoutofmemory.html, 
http://activemq.apache.org/my-producer-blocks.html,...

I have the impression that everything is working fine, but at a certain 
moment, there is a condition that makes the memory usage rise 
extensively (not a linear growth, but really a stair).

Bye,
Eric


On 04/01/2010 09:24 AM, Norman Maurer wrote:
> Hi Eric,
>
> thx again for your tests...
>
> So I think we have two independent problems here:
>
> 1) The .m64 files which are not deleted always:
>
> My guess here is that there is something wrong with processing Mail
> objects with state Null. I will investigate futher.. I also found a
> possible problem with .m64 files which get not deleted when a
> Exception is throws while processing the camel routes. I will patch
> this soon.
>
> 2) The OOM
>
> My guess here is that ActiveMQ is not freeing up memory. Not sure why yet
>
> I think 2) is the more criticial problem and should get higher
> priority to get fixed..
>
> Bye,
> Norman
>
>
>
> 2010/4/1 Eric Charles<eric.charles@u-mangate.com>:
>    
>> Hi Norman,
>>
>> Tks for the smpt-source pointer.
>> I looked at it, but finally, I think I will develop those little classes to
>> have more control on it.
>> I will put it on github.
>>
>> I launched latest trunk james yesterday evening.
>> During the night, 3 hours after the launch, the following occured:
>>
>> KahDB slower and slower...
>>
>> INFO  23:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
>> KahaDB access: cleanup took 2330
>> INFO  23:38:04,591 | james.mailetcontext | Error while storing mail.
>> org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be
>> parsed.;
>>   nested exception is:
>>     org.apache.james.imap.mailbox.StorageException: failed. Transaction
>> commit failed.;
>>   nested exception is:
>> <openjpa-1.2.1-r752877:753278 fatal store error>
>> org.apache.openjpa.persistence.RollbackException: The transaction has been
>> rolled back.  See the nested exceptions for details on the errors that
>> occurred.
>>     at
>> org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265)
>>
>>     at
>> org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:130)
>>     ... 55 more
>> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
>> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
>> 1363215207 INSERT INTO Message (id, bodyStartOctet, content, contentOctets,
>> mediaType, subType, textu
>> alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=(long) 2678, (int) 1260,
>> (byte[]) [B@7a00e70c, (long) 47887, (String) text, (String) html, (long)
>> 876]} [code=0, state=XJ001]
>>     at
>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192)
>>     at
>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57)
>>     at
>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866)
>>     at
>> org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269)
>>
>> So still a OOM exception that was shown by yet-another-component (in this
>> case, the StoreMailbox).
>>
>>
>> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
>> job).
>> All 4 files were 0 bytes.
>>
>> I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB max
>> memory).
>>
>> With the previous parameters ( -Xmx512m), the process was taking the whole
>> 512MB.
>>
>> Tks,
>> Eric
>>
>>
>> On 03/31/2010 08:26 PM, Norman Maurer wrote:
>>      
>>> Hi eric,
>>>
>>> thx for all your help, maybe smtp-source ( which is included in
>>> postfix ) can help you with stress testing.
>>>
>>> Bye
>>> Norman
>>>
>>> 2010/3/31, Eric Charles<eric.charles@u-mangate.com>:
>>>
>>>        
>>>> Hi Norman,
>>>>
>>>> There are not so much .m64 files, so even it it shows that a mail
>>>> processing didn't reach the end and may have cause a leak, I don't think
>>>> it would make crash my process so quick (between 10 and 24 hours).
>>>>
>>>> I quickly analysed a few dumps last week : there is always a class that
>>>> took 30/40% of the memory. Sometimes a activemq class, sometimes not
>>>> (don't remember the details).
>>>> Also notable, the stack trace (the place where the OOM gives problems)
>>>> varies.
>>>> Today stack was:
>>>> org.apache.camel.RuntimeCamelException:
>>>> org.apache.camel.CamelExchangeException: Error processing Exchange.
>>>> Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102,
>>>> responseRequired = true, messageId =
>>>> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination = null,
>>>> originalTransactionId = null, producerId =
>>>> ID:srv001-51032-1270011735798-2:0:24:1, destination =
>>>> queue://processor.root, transactionId = null, expiration = 0, timestamp
>>>> = 1270014325685, arrival = 0, brokerInTime = 1270014326893,
>>>> brokerOutTime = 1270014333023, correlationId = null, replyTo = null,
>>>> persistent = true, type = null, priority = 4, groupID = null,
>>>> groupSequence = 0, targetConsumerId = null, compressed = false, userID =
>>>> null, content = org.apache.activemq.util.ByteSequence@69b568d0,
>>>> marshalledProperties = null, dataStructure = null, redeliveryCounter =
>>>> 0, size = 10718, properties = null, readOnlyProperties = true,
>>>> readOnlyBody = true, droppable = false}]. Caused by:
>>>> [java.lang.OutOfMemoryError - Java heap space]
>>>>       at
>>>>
>>>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055)
>>>>       at
>>>>
>>>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154)
>>>> ...
>>>>
>>>>
>>>>
>>>> I also think the tmp files are not the main cause.
>>>>
>>>> I have now james running in eclipse and I made already made a little
>>>> trip in the code. Quite impressive since last time I looked at it.
>>>> I will try to stress james with a small smtp/pop3 client (I used
>>>> postage, but a simple class with commons-net behind) may be easier and
>>>> see what happens (eventually with eclipse profiler).
>>>>
>>>> It will be for this weekend for me.
>>>>
>>>> Bye,
>>>>
>>>> Eric
>>>>
>>>>
>>>> On 03/31/2010 07:11 PM, Norman Maurer wrote:
>>>>
>>>>          
>>>>> Hi Eric,
>>>>>
>>>>> thx for keeping us in the loop... I'm still not sure why the .m64
>>>>> files are still in the tmp folder sometimes.. But I suspect the files
>>>>> are not the cause of the OOM.
>>>>>
>>>>> I didn't get a OOM since yesterday morning (the time I deployed
>>>>> current trunk version), but just found to new .m64 files..
>>>>>
>>>>> So I'm still searching for the real cause. If nothing helps I will
>>>>> need to use a profiler to find th leak.
>>>>>
>>>>> Bye,
>>>>> Norman
>>>>>
>>>>> 2010/3/31 Eric Charles<eric.charles@u-mangate.com>:
>>>>>
>>>>>
>>>>>            
>>>>>> Hi Norman,
>>>>>>
>>>>>> I had defined the Null mailet
>>>>>>
>>>>>> <mailet match="HostIsLocal" class="Null">
>>>>>> <processor>     local-address-error</processor>
>>>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>>>> </mailet>
>>>>>>
>>>>>> but now, I use the standard config
>>>>>>
>>>>>> <mailet match="HostIsLocal" class="ToProcessor">
>>>>>> <processor>     local-address-error</processor>
>>>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>>>> </mailet>
>>>>>>
>>>>>> I still have the OOM.
>>>>>>
>>>>>> I will now deploy a fresh svn with your last commits and enable the
>>>>>> ValidRcptHandler.
>>>>>> I keep you posted with the result,
>>>>>>
>>>>>> Tks,
>>>>>> Eric
>>>>>>
>>>>>>
>>>>>> On 03/30/2010 06:49 AM, Norman Maurer wrote:
>>>>>>
>>>>>>
>>>>>>              
>>>>>>> Hi Eric,
>>>>>>>
>>>>>>> you said all the files are related to address-errors , could
you show
>>>>>>> me your address error processor config?
>>>>>>> Does the Problem still exist when you enable the ValidRcptHandler
in
>>>>>>> the smtpserver.xml file?
>>>>>>>
>>>>>>> Thx
>>>>>>> Norman
>>>>>>>
>>>>>>> 2010/3/30, Eric Charles<eric.charles@u-mangate.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>>>> Oops, no, the files are still there (only unknown@known.com).
>>>>>>>> Eric
>>>>>>>>
>>>>>>>>
>>>>>>>> On 03/29/2010 10:16 PM, Eric Charles wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                  
>>>>>>>>> Hi Norman,
>>>>>>>>>
>>>>>>>>> I just deployed your new commit with the new camel DisposeProcess.
>>>>>>>>> Good news : I don't see anymore the m64 file in tmp (well
I see 1
>>>>>>>>> file, on the second after, it is no more there, so the
dispose works
>>>>>>>>> as it should).
>>>>>>>>>
>>>>>>>>> I keep you posted with our eventual future OOM.
>>>>>>>>>
>>>>>>>>> Tks,
>>>>>>>>> Eric
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                    
>>>>>>>>>> Hi Norman,
>>>>>>>>>>
>>>>>>>>>> The .m64 are all to "unkown user" but to "well known
domain" (so
>>>>>>>>>> for
>>>>>>>>>> <unknown@known.com>).
>>>>>>>>>> They are from various size (with and without attachment).
>>>>>>>>>> They are well formed (I can open the downloaded file
with
>>>>>>>>>> thunderbird)
>>>>>>>>>>
>>>>>>>>>> However, when I sent a mail to unknown@known.com,
I don't see it in
>>>>>>>>>> the /tmp
>>>>>>>>>>
>>>>>>>>>> Tks,
>>>>>>>>>> Eric
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                      
>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>
>>>>>>>>>>> sure.. we have to find the OOM cause. What would
be interesting,
>>>>>>>>>>> could
>>>>>>>>>>> you check somehow if the .m64 files are files
which are related to
>>>>>>>>>>> successfully delivered mail ?
>>>>>>>>>>>
>>>>>>>>>>> Thx,
>>>>>>>>>>> Norman
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2010/3/29 Eric Charles<eric.charles@u-mangate.com>:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                        
>>>>>>>>>>>> Norman,
>>>>>>>>>>>>
>>>>>>>>>>>> Done. Now, we have to wait...
>>>>>>>>>>>>
>>>>>>>>>>>> I saw some created *.m64 that were removed.
>>>>>>>>>>>> But there are other ones that remains in
/tmp.
>>>>>>>>>>>>
>>>>>>>>>>>> I sometimes run a jmap (java memory map)
to produce a heap dump
>>>>>>>>>>>> and
>>>>>>>>>>>> analyse
>>>>>>>>>>>> it.
>>>>>>>>>>>> At the beginning, everything seems ok, and
often, when I come
>>>>>>>>>>>> back,
>>>>>>>>>>>> the OOM
>>>>>>>>>>>> has already occured.
>>>>>>>>>>>>
>>>>>>>>>>>> I changed the -Xmx512m to -Xmx256m to have
a quicker exception
>>>>>>>>>>>> (if
>>>>>>>>>>>> any,
>>>>>>>>>>>> let's hope not).
>>>>>>>>>>>>
>>>>>>>>>>>> The *.m64 are a clue and we should ensure
that they are removed
>>>>>>>>>>>> in
>>>>>>>>>>>> all
>>>>>>>>>>>> circumstances.
>>>>>>>>>>>> This may be the easy part, as we have some
visible clues (the
>>>>>>>>>>>> files).
>>>>>>>>>>>> We should also ensure after that there are
no other causes to the
>>>>>>>>>>>> leaks.
>>>>>>>>>>>>
>>>>>>>>>>>> Tks,
>>>>>>>>>>>> Eric
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                          
>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I found the cause for the not deleted
temporary files. Hopefully
>>>>>>>>>>>>> this
>>>>>>>>>>>>> is the cause of the OOM. Could try to
svn up and run again ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2010/3/29 Norman Maurer<norman.maurer@googlemail.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>                            
>>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> thx for the report. I see exact the
same problem today here..
>>>>>>>>>>>>>> (The
>>>>>>>>>>>>>> OOM). Didn't notice the files in
the tmp folder, but I think
>>>>>>>>>>>>>> thats
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> good pointer. I will try to debug
the problem later or
>>>>>>>>>>>>>> tomorrow.
>>>>>>>>>>>>>> But I
>>>>>>>>>>>>>> suspect you are right about the tmp
files and jms producers..
>>>>>>>>>>>>>> Did
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> run a kill -3 pid to see what threads
are active etc ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ps: Patches are welcome :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2010/3/29 Eric Charles<eric.charles@u-mangate.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>                              
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The last two weeks, I deployed
various trunk snapshots.
>>>>>>>>>>>>>>> After james running less than
1 day, I always ran into a
>>>>>>>>>>>>>>> OutOfMemory
>>>>>>>>>>>>>>> Exception.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I analysed the logs (for example
STACK TRACE 1 in annex), and
>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>> Camel complaining when trying
to deliver on JMS queue.
>>>>>>>>>>>>>>> I tried to adapt the ActiveMQ
configuration based on
>>>>>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html
but
>>>>>>>>>>>>>>> nothing
>>>>>>>>>>>>>>> helped.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I also analyzed various heap
dump :sometimes, 65% was used by
>>>>>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession,
>>>>>>>>>>>>>>> sometimes
>>>>>>>>>>>>>>> by a
>>>>>>>>>>>>>>> activemq class,...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>                             
  
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                        
>>>>>>>>>>
>>>>>>>>>>                      
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                    
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                  
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>              
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>            
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>>          
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>>
>>>        
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>      
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>    


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message