james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Danny Angus" <da...@apache.org>
Subject RE: performance figures w/ spool enabled
Date Mon, 28 Oct 2002 15:34:22 GMT
Sounds impractical but .. I bet we need to maintain an in-memory list & use it to manage
locks.
d.


> -----Original Message-----
> From: Serge Knystautas [mailto:sergek@lokitech.com]
> Sent: 28 October 2002 12:18
> To: James Developers List
> Subject: Re: performance figures w/ spool enabled
> 
> 
> One thing that was done in the DB spool was to cache the entries in the 
> spool, so you didn't have to query the database for the list of messages 
> on each attempt to get a message from the spool.  I would assume the 
> file spool could now benefit from the same caching logic.  If you think 
> about it, having to create an array of File objects (from the 
> File.list() method) would grow linearly (performance and memory) as the 
> number of messages increase.
> 
> Beyond that, you might just be stuck with just operating system woes of 
> trying to handle large numbers of files in a single directory.  If 
> that's the case, you can devise a system of creating subdirectories, but 
> that's a lot more complicated... I'd be curious how sendmail does this 
> if it is OS dependent.
> 
> -- 
> Serge Knystautas
> Loki Technologies - Unstoppable Websites
> http://www.lokitech.com
> 
> 
> Noel J. Bergman wrote:
> > Test system: 256MB RAM, Celeron 400mhz, RH 6.2, Sun JVM 1.4.1_01.
> > 
> > I ran James with sendMail (spool) enabled.  The first mailet in 
> my pipeline
> > was:
> > 
> > 	<mailet match="All" class="Null" />
> > 
> > so we immediately discard messages.
> > 
> > The test ran from 23:48 to 01:18 (90 minutes).  Total message count was
> > 35515, which means 71030 files (aggregate) were written to the spool
> > directory.  From an initial rate of ~1000 messages per minute (mpm), the
> > system slowed and (after more than an hour) stabilized at 225 
> messages per
> > minute.  This figure is pretty pathetically low (<4 mps), but 
> James didn't
> > crash and kept running.
> > 
> > My hypothesis is that performance was seriously negatively 
> degraded by the
> > file system spool.  After I shutdown the sender, the spool 
> manager started
> > to catch-up.  Starting at 01:41 (23 minutes after sending 
> stopped), the mpm
> > cleared from the spooler were:
> > 
> > 01:41:03 - 01:42:03: 27706 - 26764 =  942 / 2 = 471
> > 01:44:04 - 01:45:04: 25082 - 24218 =  864 / 2 = 432
> > 01:47:04 - 01:48:04: 22188 - 21087 = 1101 / 2 = 550
> > 01:50:04 - 01:51:04: 18658 - 17353 = 1305 / 2 = 652
> > 01:55:04 - 01:56:04: 12142 - 10662 = 1480 / 2 = 740
> > 01:59:04 - 02:00:04:  5490    3561 = 1929 / 2 = 964
> > 
> > We start with 26764 files (13382 msgs, 38% of total) still in the spool,
> > over 20 minutes after the test stopped sending messages, and 
> almost 2 hours
> > after the test started.  Notice how performance improves as 
> file count (and
> > memory use) goes down.  During the first 113 minutes, James 
> cleared an avg
> > of 195 mpm, whereas during the last 18 minutes, it cleared an avg of 744
> > mpm.
> > 
> > My hypothesis is that the higher starting rate of avg 800+ mpm 
> from a high
> > of roughly 1000 mpm, stemming from 20 SMTP handlers being faster than 10
> > spooler threads all competing for the CPU, built an initial 
> backlog.  As the
> > backlog continued to grow, the spooler continued to slow until 
> the system
> > stablized around 225 mpm.
> > 
> > If this hypothesis is correct, then in more of a burst mode, as 
> opposed to
> > steady state, James should able to keep up at closer to its 
> initial rate of
> > ~1000 mpm.  I also intend to re-test with the spool in a database, the
> > performance of which ought to scale better.
> > 
> > Anyone have other ideas?
> > 
> > Note: I am not suggesting any changes for v2.1 unless there is something
> > dramatic and easy.
> > 
> > 	--- Noel
> 
> 
> 
> --
> To unsubscribe, e-mail:   
<mailto:james-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:james-dev-help@jakarta.apache.org>


--
To unsubscribe, e-mail:   <mailto:james-dev-unsubscribe@jakarta.apache.org>
For additional commands, e-mail: <mailto:james-dev-help@jakarta.apache.org>


Mime
View raw message