james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Burrell Donkin" <robertburrelldon...@gmail.com>
Subject Re: MOM and spooling (Was: Geronimo-JAMES integration)
Date Tue, 24 Jun 2008 20:14:53 GMT
On Tue, Jun 24, 2008 at 7:40 PM, Stefano Bagnara <apache@bago.org> wrote:
> Robert Burrell Donkin ha scritto:
>>
>> On Mon, Jun 16, 2008 at 8:37 PM, Stefano Bagnara <apache@bago.org> wrote:
>>>
>>> Robert Burrell Donkin ha scritto:
>>>>
>>>> On Mon, Jun 16, 2008 at 1:01 PM, Stefano Bagnara <apache@bago.org>
>>>> wrote:
>>>>>
>>>>> Robert Burrell Donkin ha scritto:
>>>>>>
>>>>>> generally: we've talked before and the whole design around that area
>>>>>> of the system could be so much better
>>>>>>
>>>>>> technology: actually support for any distributable messaging solution
>>>>>> but i can see no reason not to use a standard
>>>>>
>>>>> I agree.
>>>>>
>>>>> FYI I had an interesting discussion with James Strachan few weeks ago,
>>>>> here:
>>>>> http://markmail.org/message/mp2pafe77efwczbb
>>>>
>>>> it's always interesting talking to james :-)
>>>>
>>>> he's definitely thinking in the same directions as me
>>>>
>>>> - robert
>>>
>>> I had no time yet to check CAMEL, and there are few issues we already
>>> found
>>> discussing the JMS solution (no way to avoid writing the full message
>>> over
>>> and over again when moving from one queue to another is my main concern),
>>> but I bookmarked the thread and camel for my next "meditation" weekend
>>> (unfortunately not so frequent lately).
>>
>> MOM works surprisingly well provided that the message sizes are small.
>> IMHO JAMES should not be attempting to load large bodies into memory.
>> instead, a limited buffer should be used initially: big enough to hold
>> the headers (and small messages). once this is filled, the message
>> should be streamed directly to storage. the message would contain the
>> headers plus a reference URL for the body.
>
> About my concern it still applies also to the scenario where only the
> "Envelope" (to be defined) is written on every queue change, but this cannot
> be avoided with the suggested approach unless we switch to non-persistent
> queues.

i think a mixture of persistent and non-persistent would probably work best.

but there should be no need to hardwire which queues are persistent:
the configuration should build this at run time

> Sometimes my JAMES deal with around a million email in a day and I had to
> "hack" some JAMES code to avoid writing to disk/jdbc on each status change
> (custom processorClass for the spoolmanager).
>
> A performance oriented approach would be to directly use the KAHA engine
>  [1] (or maybe the AMQ Message Stores [2]) o to write stuff only for
> reliability but keep using memory operations unless the queue grow too much,
> but I think this should be kept in our mind when we'll refactor the
> repository APIs (let's not close this door when we change the api)

should be easy enough if we go down the MOM route: messaging intefaces
tend to be concise which allows multiple implementations

> It would be cool if a standard SMTP=>MultipleMailets=>RemoteDelivery
> processing could be done with a single write to disk when the mail is
> accepted and no other read/write (unless memory/queues are full).

unless you're willing to risk losing messages, i don't see how this
can be done. it should be possible to just write twice: once when the
message arrives and once when processing is done.

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message