james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Charles <eric.char...@u-mangate.com>
Subject Re: OutOfMemory exception
Date Tue, 06 Apr 2010 15:41:59 GMT
Hi Norman,

james revision 930727 (mina RC1, jdk 1.6.0_18) up-and-running since 1,5 
day in prod.

jmap -dump:file=... (this morning and right now) shows 
org.apache.derby.impl.services.cache.Clock with 15% of Heap, less for 
the rest.

However, I had to redefine all accounts as IMAP in thunderbird to be 
able to read mails:
- the pop3 was working fine, and at a time-being, it hanged.
- Since there, all pop3 requests hang (thunderbird gives me a timeout 
after 2 minutes), simply saying "downloading message 1 of ..." (nothing 
in the log files)
- wireshark shows a correct conversation at the beginning, and 
afterwards many "Malformed Packets".
Maybe I could restart james, but I prefer let him work to have a better 
view on the OOM.

Tks,

Eric


On 04/02/2010 09:04 PM, Norman Maurer wrote:
> Hi Eric,
>
> I forgot to mention that one cause of the OOM could be the current
> usage of InMemoryMail. At the moment James load the whole MimeMessage
> into memory while processing the camel routes. This is really a bad
> practice in terms of performance and could maybe cause a OOM if you
> processing many big messages which are bigger (the sumary) as the
> avaible memory.
> Could this be the problem on your side ?
>
> I'm currently workin on storing the MimeMessage on an external storage
> and just stream in on demand.. So only the "meta-data" is stored in
> jms and the rest in a db / jcr etc.. I have a proof-of-concept workin
> here. But I need a bit more time to see if its the way to go..
>
> I'm also evolating the use of BlobMessages with ActiveMQ.
>
> Bye,
> Norman
>
>
> 2010/4/2 Norman Maurer<norman@apache.org>:
>    
>> Hi Eric
>>
>> so at least the files not leak anymore, unfortunaly this was the less
>> critical problem.
>> About the errors in pop3server I will have a look and see what could
>> cause this. I'm 100 % sure it's not related to the OOM.
>>
>> Any more infos related to the OOM are welcome.
>>
>> Bye
>> Norman
>>
>> 2010/4/2, Eric Charles<eric.charles@u-mangate.com>:
>>      
>>> nop. OutOfMemoryError is back.
>>>
>>> I will double check my server, jdk and environment config, but other
>>> java processes are running on the same machine without problem.
>>> I will also have a stress scenario on my local dev PC that runs fine and
>>> after, I will rerun it on my production server.
>>> Dump analysis and full profiling are also to do.
>>> Some stuff are difficult to simulate : See for example (occurs regulary)
>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>>
>>> There are no more .m64 file in /tmp
>>>
>>> Tks,
>>> Eric
>>>
>>>
>>> ERROR 16:24:53,636 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> INFO  16:24:58,061 | james.smtpserver | Successfully spooled mail
>>> Mail1270218298023-9820 from jewishnessesyh2@ecs-solar.com on
>>> 68.62.205.100 for [ric.charles@u-mangate.org, ric.charles@u-mangate.com]
>>> INFO  16:25:00,097 | org.apache.activemq.store.kahadb.MessageDatabase |
>>> Slow KahaDB access: Journal append took: 989 ms, Index update took 0 ms
>>> INFO  16:25:00,099 | james.mailetcontext | Storing mail
>>> Mail1270218298023-9820-!226071 in file://var/mail/address-error/
>>> INFO  16:25:07,699 | james.smtpserver | Successfully spooled mail
>>> Mail1270218307697-9822 from oursu31@stahelin.com on 68.62.205.100 for
>>> [ric.charles@u-mangate.be]
>>> INFO  16:25:08,723 | james.mailetcontext | Storing mail
>>> Mail1270218307697-9822-!70969 in file://var/mail/address-error/
>>> INFO  16:25:09,908 | org.apache.activemq.store.kahadb.MessageDatabase |
>>> Slow KahaDB access: Journal append took: 1184 ms, Index update took 0 ms
>>> ERROR 16:25:35,144 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> INFO  16:25:41,212 | org.apache.activemq.store.kahadb.MessageDatabase |
>>> Slow KahaDB access: cleanup took 7274
>>> INFO  16:25:55,612 | org.apache.activemq.store.kahadb.MessageDatabase |
>>> Slow KahaDB access: cleanup took 1141
>>> ERROR 16:26:42,059 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> INFO  16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
>>> Slow KahaDB access: cleanup took 58100
>>> INFO  16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
>>> Slow KahaDB access: Journal append took: 10032 ms, Index update took
>>> 52100 ms
>>> INFO  16:27:02,183 | james.smtpserver | Successfully spooled mail
>>> Mail1270218355606-9824 from inexperienceMO@toutgratuit.com on
>>> 213.163.116.78 for [angelo.biddau@u-mangate.net]
>>> WARN  16:36:27,919 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination
>>> 'processor.local-address-error' - trying to recover. Cause: Java heap space
>>> ERROR 16:43:58,281 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> WARN  16:44:23,231 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination 'processor.spam'
>>> - trying to recover. Cause: Java heap space
>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> ERROR 16:48:37,497 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> ERROR 16:49:53,684 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> ERROR 16:52:17,970 | org.apache.activemq.ActiveMQConnection | Caught an
>>> exception trying to create a JMSException for
>>> java.lang.OutOfMemoryError: Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> ERROR 16:58:05,554 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> ERROR 17:04:31,427 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> WARN  17:09:11,055 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination
>>> 'processor.local-address-error' - trying to recover. Cause: Java heap space
>>> ERROR 17:09:35,017 |
>>> org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer
>>> Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000]
>>> could not poll endpoint:
>>> pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000
>>> caused by: Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> WARN  17:09:55,361 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination 'processor.spam'
>>> - trying to recover. Cause: Java heap space
>>> WARN  17:09:57,557 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination 'processor.root'
>>> - trying to recover. Cause: Java heap space
>>> WARN  17:11:00,695 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination
>>> 'processor.local-address-error' - trying to recover. Cause: Java heap space
>>> WARN  17:22:19,706 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination
>>> 'processor.bounces' - trying to recover. Cause: Java heap space
>>> WARN  17:22:42,422 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination
>>> 'processor.transport' - trying to recover. Cause: Java heap space
>>>
>>>
>>> On 04/02/2010 09:00 AM, Norman Maurer wrote:
>>>        
>>>> Hi Eric,
>>>>
>>>> this sounds promising.... Looking forward for your next report.
>>>>
>>>> Bye
>>>> Norman
>>>>
>>>> 2010/4/2, Eric Charles<eric.charles@u-mangate.com>:
>>>>
>>>>          
>>>>> James survived the night.
>>>>> 300MB (exactly the same as when it started).
>>>>> No m64 file in tmp.
>>>>>
>>>>> I will post another update this evening.
>>>>>
>>>>> Tks,
>>>>>
>>>>> Eric
>>>>>
>>>>>
>>>>> On 04/01/2010 07:06 PM, Norman Maurer wrote:
>>>>>
>>>>>            
>>>>>> Yeah thats right.. just checout server/trunk.
>>>>>>
>>>>>> I'm looking forward to hear your feedback ;)
>>>>>>
>>>>>> Bye,
>>>>>> Norman
>>>>>>
>>>>>>
>>>>>> 2010/4/1 Eric Charles<eric.charles@u-mangate.com>:
>>>>>>
>>>>>>
>>>>>>              
>>>>>>> Hello Norman,
>>>>>>> Done :)
>>>>>>> Tks, Eric
>>>>>>>
>>>>>>> PS1 : I didn't change the config, except for the JDCBDomainList
I am
>>>>>>> using.
>>>>>>> PS2 : I only checkout
>>>>>>> http://svn.apache.org/repos/asf/james/server/trunk
>>>>>>> and
>>>>>>> not the other projects. From what I read from the poms, snapshot
>>>>>>> version
>>>>>>> of
>>>>>>> imap-mailbox, jsieve,... are taken from the central maven repository
>>>>>>> and
>>>>>>> I
>>>>>>> don't have to rebuild those projects.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>>>>     Hi Eric,
>>>>>>>>
>>>>>>>>     its me again ;) Could you please svn up to latest trunk
and try
>>>>>>>> again
>>>>>>>> ?
>>>>>>>>
>>>>>>>>     Thx,
>>>>>>>>     Norman
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                  
>>>>>>> 2010/4/1 Norman Maurer<norman.mau...@googlemail.com>:
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>>>>>     Hi Eric,
>>>>>>>>>
>>>>>>>>>     I did a stress test in the last few hours (running
in the
>>>>>>>>> background
>>>>>>>>>     here) and I was not able to get a OOM yet with
>>>>>>>>>     "?broker.useJmx=false&amp;jms.prefetchPolicy.all=1"
for the
>>>>>>>>>     jmsConnectionFactory. I sent 40000 emails with 100k
to it and 3000
>>>>>>>>>     with 5 mb.
>>>>>>>>>
>>>>>>>>>     I'm using -Xmx512m and memory usage is not getting
higher then
>>>>>>>>> 700m.
>>>>>>>>>
>>>>>>>>>     Bye.
>>>>>>>>>     Norman
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                    
>>>>>>>>                  
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>              
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>            
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>          
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>        
>>      
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
>
>    


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message