james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Charles <eric.char...@u-mangate.com>
Subject Re: JPAStressTest with derby, h2 and mysql
Date Mon, 03 May 2010 17:32:48 GMT
Hi Norman,

I've set ulimit -n 65535 (default was 1024), used -Xmx512m (cfr your 
last commit) and connectionBacklog=900

lsof -p <pid>| wc -l gives around 550 (stable).
No issue after stop (test with embedded Derby).

Disk swapping and exceptions came from the 64m limit if you don't set 
-Xmx (james is not that memory hungry beast, but 64 is really small).
Will try on real server later this week to rise the throughput.

Tks,

Eric

On 05/02/2010 06:37 PM, Norman Maurer wrote:
> Ups I mean
>
>        <connectionBacklog>200</connectionBacklog>
>
> in smtpserver.xml
>
> Bye,
> Norman
>
> 2010/5/2 Eric Charles<eric.charles@u-mangate.com>:
>    
>> Hi Norman,
>> What do you mean with "...backlock in smtpserver.log" ?
>> Tks,
>> Eric
>>
>> On 05/02/2010 01:15 PM, Norman Maurer wrote:
>>      
>>> Hi Eric,
>>>
>>> for such high load you will need to adjust a few things:
>>>
>>> 1) set a higher ulimit. By default its 1024 on linux which is prolly
>>> not enough ( I use fbsd which use 11095 by default)
>>> 2) set a higher connection backlock in smtpserver.log
>>>
>>>
>>> Bye,
>>> Norman
>>>
>>>
>>> 2010/5/2 Eric Charles<eric.charles@u-mangate.com>:
>>>
>>>        
>>>> Hi Norman,
>>>>
>>>> Trying to simulate a more real-life test case, I developed a small bomber
>>>> class that launches parallel threads. Each thread sends mails with
>>>> attachements (with small pause between each mail sending).
>>>>
>>>> I tested with embedded derby with JDBCDomainList.
>>>>
>>>> Depending on the parameters (10 to 100 parallel threads, 5 ms to 100 ms
>>>> pause, 1KB to 100KB attachement), I could have on my dev laptop 5 to 10
>>>> mails per second being processed.
>>>>
>>>> The good think is that I have no exception at all during stable phase.
>>>>
>>>> However, I was not able to have more mails being processed on my latptop
>>>> due
>>>> to disk access. The CPU and memory are really not charged (20 - 30 %),
>>>> but
>>>> disk is really working non-stop.
>>>> This is probably the price to pay for activemq and the kahadb.
>>>>
>>>> Complete scenario with unstable phase:
>>>> - Bomb 5 minutes
>>>> - Stop bomb.
>>>> - Still some mails are spooled for a while which is quite normal.
>>>> - When all mails are spooled, disk still runs crazy for minutes, console
>>>> showing a few messages "Slow KahaDB access: cleanup took 684"
>>>> - rebombing : org.apache.camel.CamelExecutionException: Exception
>>>> occurred
>>>> during execution on the exchange: Exchange[Message:
>>>> org.apache.james.core.MailImpl@16863f2] Caused by: java.io.IOException:
>>>> Too
>>>> many open files - error message in the bomber
>>>> - Server console show still a few mails being spooled.
>>>> - After, server does not respond on socket - no more exception.
>>>>
>>>> Disk congestion may come from my disk.
>>>> I will setup a server environment with good disk, and will rerun the test
>>>> in
>>>> a few days.
>>>> A database on a separate server could also help.
>>>>
>>>> Btw, what should we consider as acceptable/needed as load (xxx mails from
>>>> yyy parallel clients) ?
>>>>
>>>> Tks,
>>>>
>>>> Eric
>>>>
>>>>
>>>>
>>>> On 05/01/2010 09:25 AM, Norman Maurer wrote:
>>>>
>>>>          
>>>>> About 6000, then I run out of heap space..
>>>>>
>>>>> Can you maybe try to deploy it with james and bomb it via smtp ? I was
>>>>> not able to see any exception.
>>>>>
>>>>> Thx,
>>>>> Norman
>>>>>
>>>>>
>>>>> 2010/5/1 Eric Charles<eric.charles@u-mangate.com>:
>>>>>
>>>>>
>>>>>            
>>>>>> Hi Norman,
>>>>>>
>>>>>> Similar exceptions with properties.put("openjpa.LockTimeout", "30000");
>>>>>>
>>>>>> I think the environment simply can not follow the load (more than
one
>>>>>> mail
>>>>>> sent each ms).
>>>>>> Bootleneck can be anywhere (connection creation, database lock,
>>>>>> openjpa,...).
>>>>>> This is why I put a sleep between each mail sending.
>>>>>> Real servers will better hold the pressure.
>>>>>>
>>>>>> So I think I'm limited by my environment : it does not allow me to
have
>>>>>> a
>>>>>> really heavy load to validate that IMAP-137 is resolved.
>>>>>> Well, it is probably, as before you commits, I directly had the
>>>>>> duplicate
>>>>>> key exception.
>>>>>>
>>>>>> How much mails can you hold on your env with h2 inmemory ?
>>>>>>
>>>>>> Tks,
>>>>>>
>>>>>> Eric
>>>>>>
>>>>>>
>>>>>> On 05/01/2010 09:08 AM, Norman Maurer wrote:
>>>>>>
>>>>>>
>>>>>>              
>>>>>>> Hi Eric,
>>>>>>>
>>>>>>> thx for your tests. The order of the uid is caused by the threads,
>>>>>>> thats nothing to worry about.
>>>>>>>
>>>>>>> About the timeout, can you try to add this to the test case:
>>>>>>>
>>>>>>> properties.put("openjpa.LockTimeout", "30000");
>>>>>>>
>>>>>>> And keep in mind that we don't use any connection pooling in
the test
>>>>>>> case so the performance should be better when use it in deployment.
>>>>>>>
>>>>>>> Bye,
>>>>>>> Norman
>>>>>>>
>>>>>>>
>>>>>>> 2010/5/1 Eric Charles<eric.charles@u-mangate.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I just tested the JPAStressTest (for IMAP-137 JPA fails to
persist
>>>>>>>> MailboxMembership Entity on heavy load) Norman committed
yesterday.
>>>>>>>>
>>>>>>>> I made some tests towards derby (memory and embedded) , h2
(memory
>>>>>>>> and
>>>>>>>> embedded) and mysql with following configs (and adding the
needed
>>>>>>>> jdbc
>>>>>>>> drivers in pom.xml):
>>>>>>>>
>>>>>>>>         // Derby Memory
>>>>>>>>         properties.put("openjpa.ConnectionDriverName",
>>>>>>>> org.apache.derby.jdbc.EmbeddedDriver.class.getName());
>>>>>>>>         properties.put("openjpa.ConnectionURL",
>>>>>>>> "jdbc:derby:memory:derbyimap;create=>>>>>>>
>>>>>>>>         // Derby Embedded
>>>>>>>> //        properties.put("openjpa.ConnectionDriverName",
>>>>>>>> org.apache.derby.jdbc.EmbeddedDriver.class.getName());
>>>>>>>> //        properties.put("openjpa.ConnectionURL",
>>>>>>>> "jdbc:derby:derbyimap;create=r=root;password=root");
>>>>>>>>
>>>>>>>>         // H2 Memory
>>>>>>>> //        properties.put("openjpa.ConnectionDriverName",
>>>>>>>> org.h2.Driver.class.getName());
>>>>>>>> //        properties.put("openjpa.ConnectionURL",
>>>>>>>> "jdbc:h2:mem:h2imap;DB_CLOSE_DELAY=>>>
>>>>>>>>         // H2 Embedded
>>>>>>>> //        properties.put("openjpa.ConnectionDriverName",
>>>>>>>> org.h2.Driver.class.getName());
>>>>>>>> //        properties.put("openjpa.ConnectionURL",
>>>>>>>> "jdbc:h2:~/h2/h2imap;USER=SWORD=root");
>>>>>>>>
>>>>>>>>         //Mysql
>>>>>>>> //        properties.put("openjpa.ConnectionDriverName",
>>>>>>>> com.mysql.jdbc.Driver.class.getName());
>>>>>>>> //        properties.put("openjpa.ConnectionURL",
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> "jdbc:mysql://localhost:3306/msqlimap?user=sword=root&createDatabaseIfNotExist=true&amp;useUnicode=true&amp;characterEncoding=utf-8");
>>>>>>>>
>>>>>>>>
>>>>>>>> I first changed the maximum number of messages from 1.000
to 100.000
>>>>>>>> and
>>>>>>>> saw
>>>>>>>> on my linux dev PC (james server, stresstest and databases
all
>>>>>>>> running
>>>>>>>> on
>>>>>>>> same dev PC):
>>>>>>>>
>>>>>>>> Derby: after many mails sending:
>>>>>>>> java.io.IOException: Too many open files
>>>>>>>> ...
>>>>>>>> Exception in thread "pool-1-thread-1044"
>>>>>>>> java.lang.NoClassDefFoundError:
>>>>>>>> org/apache/james/imap/api/display/HumanReadableText
>>>>>>>> ...
>>>>>>>>
>>>>>>>> H2: after 400 mails:
>>>>>>>> java.io.IOException: Too many open files
>>>>>>>> ...
>>>>>>>> Caused by:<openjpa-1.2.2-r422266:898935 nonfatal general
error>
>>>>>>>> org.apache.openjpa.persistence.PersistenceException: Timeout
trying
>>>>>>>> to
>>>>>>>> lock
>>>>>>>> table MEMBERSHIP [50200-79] {prepstmnt 406576
>>>>>>>> INSERT INTO Membership (mailboxId, uid, answered, deleted,
draft,
>>>>>>>>         flagged, internalDate, recent, seen, size, MESSAGE_ID)
>>>>>>>>     VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
>>>>>>>> [params=, (long) 49, (int) 0, (int) 0, (int) 0, (int) 0,
>>>>>>>> (Timestamp)
>>>>>>>> 2010-05-01 08:37:10.293, (int) 0, (int) 0, (int) 25, (long)
4151]}
>>>>>>>> [codeP200, state=
>>>>>>>> ...
>>>>>>>>
>>>>>>>> MySql : after +/- 1000 (always around 1002 and 1007) mails
:
>>>>>>>> Exception in thread "pool-1-thread-84"<openjpa-1.2.2-r422266:898935
>>>>>>>> nonfatal user error>
>>>>>>>>   org.apache.openjpa.persistence.NoResultException:
>>>>>>>> The
>>>>>>>> query on candidate type "class
>>>>>>>> org.apache.james.imap.jpa.mail.model.JPAMailbox" with filter
"SELECT
>>>>>>>> mailbox
>>>>>>>> FROM Mailbox mailbox WHERE mailbox.mailboxId =m" was configured
to
>>>>>>>> have a unique result, but no instance matched the query.
>>>>>>>> ....
>>>>>>>> org.apache.james.imap.mailbox.MailboxNotFoundException: Mailbox
>>>>>>>> '#mail.INBOX' not found.
>>>>>>>> ...
>>>>>>>>
>>>>>>>> 3 different databases, 3 different behaviour/exceptions.
>>>>>>>>
>>>>>>>> Running 3 components (james server, stresstest, database)
on my
>>>>>>>> "not-so-strong-PC" may be an issue.
>>>>>>>>
>>>>>>>> To make the tests successful, I added a Thread.sleep(5) (15
ms for
>>>>>>>> embedded
>>>>>>>> derby and h2) between each mail sending.
>>>>>>>>
>>>>>>>>
>>>>>>>> I suppose we can say that in a real environment, the load
will never
>>>>>>>> be
>>>>>>>> that
>>>>>>>> heavy, and that real server will perform better.
>>>>>>>>
>>>>>>>>
>>>>>>>> I also saw that the uid order is not respected:
>>>>>>>>
>>>>>>>> Append message with uidb58
>>>>>>>> Append message with uidb59
>>>>>>>> Append message with uidb56
>>>>>>>> Append message with uidb57
>>>>>>>>
>>>>>>>> This could come from a print mismatch, and not from uid being
>>>>>>>> generated
>>>>>>>> in a
>>>>>>>> bad order.
>>>>>>>>
>>>>>>>> Tks in advance for your feedbacks, if any,
>>>>>>>>
>>>>>>>>
>>>>>>>> Eric
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                  
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>              
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>            
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>>          
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>        
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>      
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>    


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message