james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joachim Draeger ...@joachim-draeger.de>
Subject Re: IMAP Draft: Cluster
Date Thu, 06 Jul 2006 07:38:50 GMT

Hi Bernd,

> > Imap is quite resource intensive. 
> 
> Is it? Why?
> How does IMAP resource consumption compare to already existing James 
> parts like POP3 and UserRepositories?

The normal SMTP/POP life-cycle is 1. delivering a message 2. retrieve by
pop3, store on users hard-disk, delete from server. The user will sort
and archive his messages on his own computer. 
With IMAP all messages stay on the server. The client is not required to
cache anything. The user will browse through the mailboxes fetch headers
and content, perform searches, copy messages.
With POP3 mailboxes with the size of a GB are very unlikely. 
IMAP is designed to keep a TCP connection open all the time do be
informed when new mail arrives. Clients are even opening multiple
concurrent connections when browsing through mailboxes.
In a worst case, when nobody is ill or on holidays, you have at least
one open connection per user all the time. 

> I would have guessed think that the resource consumption is not 
> depending on the protocol in the first place, but on the number of 
> users, the number of transmitted emails and the size of those emails.

Yes, but on IMAP the user is dealing with the same message several
times.
Of course it's not the protocol itself but the way it is used.

> This requires to share the load
> > between multiple hosts. Another goal is being fault tolerant and to have
> > a fail-over strategy. This means not having 99.999% availability. This
> > could be done using a cluster capable RDBMS. At the best this would mean
> > that you can setup your IMAP servers like accessing one single database.
> 
> Are you still talking about James at this point or some specialized IMAP 
> server apart from James?

For accessing a DB cluster you don't need any specialization. Any
application that uses JDBC should be able to do it quite out of the box.

> Clustering at DB level would mean to not support dbfile, file and other 
>   types of repositories.

Clustering at DB level would just mean following some rules in the
implementation. DBMS are designed to be accessed simultaneously from
multiple hosts.
dbfile mmh, good point. I just made a few thoughts. There are also
various solutions to access a network file-system without a single point
of failure. And it is promising the best scalability for really high
volume. 100 GB in a file-system is no problem 100 GB in a DB might be a
nightmare. :-) 
The simultaneous access could be managed through a central DB. 

Reading a message:

1. acquire a read-only lock to the row representing the message
2. perform the read from network file system
3. release the lock

Concurrent read operation would be no problem. Even setting flags while
another is reading is possible when they are stored in a different
table. This includes the \Deleted flag. Only expunging deleted messages
will be blocked until everyone has finished reading. 
At the moment locking in the JDBCMailRepository is done by a local
HashTable and has some weak points in its implementation.

For file-system only based repository I don't see a chance for being
imap capable and cluster-able  at the same time.
There has to be one instance that has to assign the uids. 
Even when doing it without uid and IMAP capability accessing a
file-based store from multiple hosts has always drawbacks.
There is no solution for reliable locking. Even Maildir has its weak
points when under heavy access. Trying to delete a file another one is
reading means starting a timer and multiple retries. If someone else is
modifying flags the file-name changes and you have to list the whole
directory again to see if it is still there...

Former Maildir proposals had such scary ideas like only being able to
deliver one email per second and process to a mailbox, retrying every 2
seconds up to 24 hours. 

Joachim



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message