james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danny Angus <danny.an...@gmail.com>
Subject Re: Summer of Code - JAMES Fastfail and Clustering - My observation
Date Fri, 03 Jun 2005 09:19:01 GMT

> Handler receives data, and forms
> approriate commands and call doXXX to handle these commands. One
> possible clustering approach here is to distribute these
> commands to different processes (in different JVM as well).

We currently assign each socket conenction to a handler and each
handler is an independant thread.
We could distribute at this level, handing off connections to cluster
members for handling, but commands in each session are sequential and
the session is stateful, meaning that if we hand commands off we would
have to also ensure that session state is synchronised and that
command sequence rules aren't broken by clustering accidentally
multi-tasking each session.

> For the mail processing pipeline in which LinearProcessor processes mail
> from Spool by letting it down to the chain of matcher
> and mailet pairs. I suspect that this chain is different for different
> James server. It means that we can try to keep the state of this
> chain among the cluster.

The chain (we tend to call it the processor pipeline) would be
different for each install, if we hand a clustered install I would
expect the chain to be the same across all instances in the cluster.

IMHO the best place to start would be to cluster the spoolmanager
(which runs the processor pipeline) this would allow many de-queueing
processes to process many mails simultaneously. This would result in a
fairly constant cost per-message scaleable capacity, allow the use of
commodity box clusters, and provide resilience.

To make the whole set-up resilient and HA you'd need to add protocol clustering.

I think the main tasks would be:

1/ need to have resilient, scaleable shared storage and distributed locking
2/ need to have some kind of cluster management to synchronise configuration.

But thats just my 2c


> Could any developers verify my observations and give some comments ? I
> am really interested in implementing clustering feature
> for James.
> Thanks all.
> Danny Angus wrote:
> >I suggest you subscribe to server-dev@james.apache.org
> >and get involved in the discussions there.
> >
> >I don't know anything about the clustering proposal.
> >I think the google summer of code is intended to be a three month
> >"engagement" and I suspect that three months full-time is more than
> >plenty time to accomplish either of these two goals.
> >
> >In my opinion (There may be some disagreement about where we're going
> >with this!) Fast Fail is about extending the SMTP protocol to include
> >validations which allow the server to terminate the conversation
> >before it wastes valuable resources downloading undeliverable, spam or
> >malicious email, or mail which violates some kind of other policy
> >(maximum size, user mailbox size that kind of thing).
> >
> >The James project want to implement this is a way which complies with
> >our architectural policy.
> >
> >Although the policy isn't written down it is clear from looking at
> >James' code that this change will have to be implemented such that the
> >protocol handlers are re-desinged in a modular way, and in a manner
> >sugested by an OO analysis of the general process of handling
> >protocols of this class. (ascii stateful request/response over tcp/ip)
> >There are one or two options here, and one part of the task is to try
> >to merge everyones requirements and arrive at an approach which gains
> >the consensus of the group. The next part is to design some kind of
> >simple API (and implement it in James) which will allow "rules" to be
> >developed and applied declaratively through configuration, in keeping
> >with the way the rest of James functionality is defined. This will
> >involve identifying the set of data which the server should make
> >available to the rules.
> >The third and final part is to then implement some of the required
> >rules and release it for testing by the community.
> >
> >
> >
> >d.
> >
> >
> >On 02/06/05, Justin Berka <jberka@andrew.cmu.edu> wrote:
> >
> >
> >>Danny,
> >>        I'm browsing projects ideas for summer of code, and the fastfail and
> >>clustering for JAMES both caught my eye. Could you tell me a bit more
> >>about these, what they would require, how much time they would take,
> >>etc?
> >>
> >>        Thanks,
> >>                        Justin
> >>
> >>
> >>
> >>
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> >For additional commands, e-mail: server-dev-help@james.apache.org
> >
> >
> >
> >
> >
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org

To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

View raw message