james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Norman Maurer <norman.mau...@googlemail.com>
Subject Re: JPAStressTest with derby, h2 and mysql
Date Mon, 03 May 2010 17:34:24 GMT
Hi Eric,

thx for all your help and tests :)

Bye,
Norman


2010/5/3 Eric Charles <eric.charles@u-mangate.com>:
> Hi Norman,
>
> I've set ulimit -n 65535 (default was 1024), used -Xmx512m (cfr your last
> commit) and connectionBacklog=900
>
> lsof -p <pid>| wc -l gives around 550 (stable).
> No issue after stop (test with embedded Derby).
>
> Disk swapping and exceptions came from the 64m limit if you don't set -Xmx
> (james is not that memory hungry beast, but 64 is really small).
> Will try on real server later this week to rise the throughput.
>
> Tks,
>
> Eric
>
> On 05/02/2010 06:37 PM, Norman Maurer wrote:
>>
>> Ups I mean
>>
>>       <connectionBacklog>200</connectionBacklog>
>>
>> in smtpserver.xml
>>
>> Bye,
>> Norman
>>
>> 2010/5/2 Eric Charles<eric.charles@u-mangate.com>:
>>
>>>
>>> Hi Norman,
>>> What do you mean with "...backlock in smtpserver.log" ?
>>> Tks,
>>> Eric
>>>
>>> On 05/02/2010 01:15 PM, Norman Maurer wrote:
>>>
>>>>
>>>> Hi Eric,
>>>>
>>>> for such high load you will need to adjust a few things:
>>>>
>>>> 1) set a higher ulimit. By default its 1024 on linux which is prolly
>>>> not enough ( I use fbsd which use 11095 by default)
>>>> 2) set a higher connection backlock in smtpserver.log
>>>>
>>>>
>>>> Bye,
>>>> Norman
>>>>
>>>>
>>>> 2010/5/2 Eric Charles<eric.charles@u-mangate.com>:
>>>>
>>>>
>>>>>
>>>>> Hi Norman,
>>>>>
>>>>> Trying to simulate a more real-life test case, I developed a small
>>>>> bomber
>>>>> class that launches parallel threads. Each thread sends mails with
>>>>> attachements (with small pause between each mail sending).
>>>>>
>>>>> I tested with embedded derby with JDBCDomainList.
>>>>>
>>>>> Depending on the parameters (10 to 100 parallel threads, 5 ms to 100
ms
>>>>> pause, 1KB to 100KB attachement), I could have on my dev laptop 5 to
10
>>>>> mails per second being processed.
>>>>>
>>>>> The good think is that I have no exception at all during stable phase.
>>>>>
>>>>> However, I was not able to have more mails being processed on my
>>>>> latptop
>>>>> due
>>>>> to disk access. The CPU and memory are really not charged (20 - 30 %),
>>>>> but
>>>>> disk is really working non-stop.
>>>>> This is probably the price to pay for activemq and the kahadb.
>>>>>
>>>>> Complete scenario with unstable phase:
>>>>> - Bomb 5 minutes
>>>>> - Stop bomb.
>>>>> - Still some mails are spooled for a while which is quite normal.
>>>>> - When all mails are spooled, disk still runs crazy for minutes,
>>>>> console
>>>>> showing a few messages "Slow KahaDB access: cleanup took 684"
>>>>> - rebombing : org.apache.camel.CamelExecutionException: Exception
>>>>> occurred
>>>>> during execution on the exchange: Exchange[Message:
>>>>> org.apache.james.core.MailImpl@16863f2] Caused by: java.io.IOException:
>>>>> Too
>>>>> many open files - error message in the bomber
>>>>> - Server console show still a few mails being spooled.
>>>>> - After, server does not respond on socket - no more exception.
>>>>>
>>>>> Disk congestion may come from my disk.
>>>>> I will setup a server environment with good disk, and will rerun the
>>>>> test
>>>>> in
>>>>> a few days.
>>>>> A database on a separate server could also help.
>>>>>
>>>>> Btw, what should we consider as acceptable/needed as load (xxx mails
>>>>> from
>>>>> yyy parallel clients) ?
>>>>>
>>>>> Tks,
>>>>>
>>>>> Eric
>>>>>
>>>>>
>>>>>
>>>>> On 05/01/2010 09:25 AM, Norman Maurer wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> About 6000, then I run out of heap space..
>>>>>>
>>>>>> Can you maybe try to deploy it with james and bomb it via smtp ?
I was
>>>>>> not able to see any exception.
>>>>>>
>>>>>> Thx,
>>>>>> Norman
>>>>>>
>>>>>>
>>>>>> 2010/5/1 Eric Charles<eric.charles@u-mangate.com>:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Hi Norman,
>>>>>>>
>>>>>>> Similar exceptions with properties.put("openjpa.LockTimeout",
>>>>>>> "30000");
>>>>>>>
>>>>>>> I think the environment simply can not follow the load (more
than one
>>>>>>> mail
>>>>>>> sent each ms).
>>>>>>> Bootleneck can be anywhere (connection creation, database lock,
>>>>>>> openjpa,...).
>>>>>>> This is why I put a sleep between each mail sending.
>>>>>>> Real servers will better hold the pressure.
>>>>>>>
>>>>>>> So I think I'm limited by my environment : it does not allow
me to
>>>>>>> have
>>>>>>> a
>>>>>>> really heavy load to validate that IMAP-137 is resolved.
>>>>>>> Well, it is probably, as before you commits, I directly had the
>>>>>>> duplicate
>>>>>>> key exception.
>>>>>>>
>>>>>>> How much mails can you hold on your env with h2 inmemory ?
>>>>>>>
>>>>>>> Tks,
>>>>>>>
>>>>>>> Eric
>>>>>>>
>>>>>>>
>>>>>>> On 05/01/2010 09:08 AM, Norman Maurer wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Hi Eric,
>>>>>>>>
>>>>>>>> thx for your tests. The order of the uid is caused by the
threads,
>>>>>>>> thats nothing to worry about.
>>>>>>>>
>>>>>>>> About the timeout, can you try to add this to the test case:
>>>>>>>>
>>>>>>>> properties.put("openjpa.LockTimeout", "30000");
>>>>>>>>
>>>>>>>> And keep in mind that we don't use any connection pooling
in the
>>>>>>>> test
>>>>>>>> case so the performance should be better when use it in deployment.
>>>>>>>>
>>>>>>>> Bye,
>>>>>>>> Norman
>>>>>>>>
>>>>>>>>
>>>>>>>> 2010/5/1 Eric Charles<eric.charles@u-mangate.com>:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I just tested the JPAStressTest (for IMAP-137 JPA fails
to persist
>>>>>>>>> MailboxMembership Entity on heavy load) Norman committed
yesterday.
>>>>>>>>>
>>>>>>>>> I made some tests towards derby (memory and embedded)
, h2 (memory
>>>>>>>>> and
>>>>>>>>> embedded) and mysql with following configs (and adding
the needed
>>>>>>>>> jdbc
>>>>>>>>> drivers in pom.xml):
>>>>>>>>>
>>>>>>>>>        // Derby Memory
>>>>>>>>>        properties.put("openjpa.ConnectionDriverName",
>>>>>>>>> org.apache.derby.jdbc.EmbeddedDriver.class.getName());
>>>>>>>>>        properties.put("openjpa.ConnectionURL",
>>>>>>>>> "jdbc:derby:memory:derbyimap;create=>>>>>>>
>>>>>>>>>        // Derby Embedded
>>>>>>>>> //        properties.put("openjpa.ConnectionDriverName",
>>>>>>>>> org.apache.derby.jdbc.EmbeddedDriver.class.getName());
>>>>>>>>> //        properties.put("openjpa.ConnectionURL",
>>>>>>>>> "jdbc:derby:derbyimap;create=r=root;password=root");
>>>>>>>>>
>>>>>>>>>        // H2 Memory
>>>>>>>>> //        properties.put("openjpa.ConnectionDriverName",
>>>>>>>>> org.h2.Driver.class.getName());
>>>>>>>>> //        properties.put("openjpa.ConnectionURL",
>>>>>>>>> "jdbc:h2:mem:h2imap;DB_CLOSE_DELAY=>>>
>>>>>>>>>        // H2 Embedded
>>>>>>>>> //        properties.put("openjpa.ConnectionDriverName",
>>>>>>>>> org.h2.Driver.class.getName());
>>>>>>>>> //        properties.put("openjpa.ConnectionURL",
>>>>>>>>> "jdbc:h2:~/h2/h2imap;USER=SWORD=root");
>>>>>>>>>
>>>>>>>>>        //Mysql
>>>>>>>>> //        properties.put("openjpa.ConnectionDriverName",
>>>>>>>>> com.mysql.jdbc.Driver.class.getName());
>>>>>>>>> //        properties.put("openjpa.ConnectionURL",
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> "jdbc:mysql://localhost:3306/msqlimap?user=sword=root&createDatabaseIfNotExist=true&amp;useUnicode=true&amp;characterEncoding=utf-8");
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I first changed the maximum number of messages from 1.000
to
>>>>>>>>> 100.000
>>>>>>>>> and
>>>>>>>>> saw
>>>>>>>>> on my linux dev PC (james server, stresstest and databases
all
>>>>>>>>> running
>>>>>>>>> on
>>>>>>>>> same dev PC):
>>>>>>>>>
>>>>>>>>> Derby: after many mails sending:
>>>>>>>>> java.io.IOException: Too many open files
>>>>>>>>> ...
>>>>>>>>> Exception in thread "pool-1-thread-1044"
>>>>>>>>> java.lang.NoClassDefFoundError:
>>>>>>>>> org/apache/james/imap/api/display/HumanReadableText
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>> H2: after 400 mails:
>>>>>>>>> java.io.IOException: Too many open files
>>>>>>>>> ...
>>>>>>>>> Caused by:<openjpa-1.2.2-r422266:898935 nonfatal general
error>
>>>>>>>>> org.apache.openjpa.persistence.PersistenceException:
Timeout trying
>>>>>>>>> to
>>>>>>>>> lock
>>>>>>>>> table MEMBERSHIP [50200-79] {prepstmnt 406576
>>>>>>>>> INSERT INTO Membership (mailboxId, uid, answered, deleted,
draft,
>>>>>>>>>        flagged, internalDate, recent, seen, size,
MESSAGE_ID)
>>>>>>>>>    VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
>>>>>>>>> [params=, (long) 49, (int) 0, (int) 0, (int) 0, (int)
0,
>>>>>>>>> (Timestamp)
>>>>>>>>> 2010-05-01 08:37:10.293, (int) 0, (int) 0, (int) 25,
(long) 4151]}
>>>>>>>>> [codeP200, state=
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>> MySql : after +/- 1000 (always around 1002 and 1007)
mails :
>>>>>>>>> Exception in thread "pool-1-thread-84"<openjpa-1.2.2-r422266:898935
>>>>>>>>> nonfatal user error>
>>>>>>>>>  org.apache.openjpa.persistence.NoResultException:
>>>>>>>>> The
>>>>>>>>> query on candidate type "class
>>>>>>>>> org.apache.james.imap.jpa.mail.model.JPAMailbox" with
filter
>>>>>>>>> "SELECT
>>>>>>>>> mailbox
>>>>>>>>> FROM Mailbox mailbox WHERE mailbox.mailboxId =m" was
configured to
>>>>>>>>> have a unique result, but no instance matched the query.
>>>>>>>>> ....
>>>>>>>>> org.apache.james.imap.mailbox.MailboxNotFoundException:
Mailbox
>>>>>>>>> '#mail.INBOX' not found.
>>>>>>>>> ...
>>>>>>>>>
>>>>>>>>> 3 different databases, 3 different behaviour/exceptions.
>>>>>>>>>
>>>>>>>>> Running 3 components (james server, stresstest, database)
on my
>>>>>>>>> "not-so-strong-PC" may be an issue.
>>>>>>>>>
>>>>>>>>> To make the tests successful, I added a Thread.sleep(5)
(15 ms for
>>>>>>>>> embedded
>>>>>>>>> derby and h2) between each mail sending.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I suppose we can say that in a real environment, the
load will
>>>>>>>>> never
>>>>>>>>> be
>>>>>>>>> that
>>>>>>>>> heavy, and that real server will perform better.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I also saw that the uid order is not respected:
>>>>>>>>>
>>>>>>>>> Append message with uidb58
>>>>>>>>> Append message with uidb59
>>>>>>>>> Append message with uidb56
>>>>>>>>> Append message with uidb57
>>>>>>>>>
>>>>>>>>> This could come from a print mismatch, and not from uid
being
>>>>>>>>> generated
>>>>>>>>> in a
>>>>>>>>> bad order.
>>>>>>>>>
>>>>>>>>> Tks in advance for your feedbacks, if any,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Eric
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message