james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Charles <e...@apache.org>
Subject Re: IMAP Fetch OOM [WAS: Re: ActiveMQ the cause?]
Date Thu, 14 Oct 2010 12:09:35 GMT
Total Retained Heap: 341.033.584
where top is OpenJPAMailboxManager: 266.079.320 which references 
266.068.064 of DelegatingMailboxListeners.

In DelegatingMailboxListerners, there is the Map<MailboxPath, 
List<MailboxListener>> listeners.

I can see there the 25 map entries, where
- first one is 105.017.544
- second one is 71.025
-...
for the total of 266.068.064

The first map entry for the MailboxPath being of large subfolder of my 
inbox has
- 301 instances of UidToMsnConverter in a list for 87.229.424
- 200 instances of UidToMsnConverter in a list (which is it self in a 
map) for 17.788.032

Each UidToMsnConverter contains 2 TreeMap (msnToUid and uidToMsn), each 
having 1.037.984.

Have to replay it, but could we have one instance of UidToMsnConverter 
per session to spare some memory?

Tks,

Eric


On 14/10/2010 13:44, Norman wrote:
>  Hi Eric,
>
> how big was the memory usage of the Map which holds the MailboxPath ? 
> Just to get an idea if we need to improve it.
>
> Bye,
> Norman
>
>
> Am 14.10.2010 13:35, schrieb Eric Charles:
>> Hi Norman,
>>
>> Yes, it solved oom atm.
>> I was not able to reproduce it while fetching mails from 2 clients.
>> However, I saw very fast fetching at start, slowing down at the end.
>> So, I fear that launching many clients in // to my mailbox would 
>> still slow it, and probably give oom (but it's true that I use the 
>> default -Xmx512M).
>>
>> I took a mem dump, and at first look, jpaheader,... object do not eat 
>> the memory anymore.
>> The largest memory usage is OpenJpaManager which references a Map 
>> with many MailboxPath.
>>
>> Yes, gc logs should be activated, and we should also find the best 
>> configs (there's many one).
>> I also would like to test the G1 gc that seems more stable now than 
>> it was last year.
>>
>> But i will stop monitoring/analyzing gc/memory for now to concentrate 
>> on M1 release.
>> We should take this between M1 and M2, especially with automated 
>> stress tests on all protocols (stmp/imap/pop3/...).
>>
>> Tks,
>>
>> Eric
>>
>>
>> On 14/10/2010 13:12, Norman wrote:
>>>  Hi Eric,
>>>
>>> so the patch solves the OOM for you ? Did you have a look at the 
>>> heap (via jconsole;) ) while do the action ? Would be interesting to 
>>> see how the GC kicks in..
>>>
>>> And yes I think we can have it in M1 ;)
>>>
>>> Bye
>>> Norman
>>>
>>>
>>> Am 14.10.2010 12:10, schrieb Eric Charles:
>>>> Hi Norman,
>>>>
>>>> I tested with the patch you sent me off-ml, and it rebuilding index 
>>>> on a large index goes 10 ways faster :)
>>>> I still didn't succeed to get the oom synchronizing.
>>>>
>>>> Do you think we could have this patch in the coming imap 0.2-M1 
>>>> release?
>>>>
>>>> For later releases, we could still talk about ways to enhance the 
>>>> FETCH performance (full streaming solution for example).
>>>>
>>>> Tks,
>>>>
>>>> Eric
>>>>
>>>>
>>>> On 12/10/2010 10:27, Eric Charles wrote:
>>>>> - read "raw mail" instead of "raw content"
>>>>> - to have added value, "streaming from store to socket via 
>>>>> listener" goes with "store raw mail".
>>>>>
>>>>>
>>>>> On 12/10/2010 10:10, Eric Charles wrote:
>>>>>> Hi Norman,
>>>>>>
>>>>>> Yes, Iterator<MessageResult> getMessages(MessageRange set,
...) 
>>>>>> is really the place where we have to  act to reduce the memory 
>>>>>> consumption.
>>>>>>
>>>>>> I also though about some ways to better scale.
>>>>>> I didn't really think to a batch way, but this can help, even if

>>>>>> we will have to go to the repository (database for example) 10 
>>>>>> times in your example.
>>>>>> Also I'm not sure the gc will act before the end of the batch 
>>>>>> processing (depends on the gc config), so you could still have 
>>>>>> memory growing very fast to serve a huge fetch, without being 
>>>>>> really garbaged.
>>>>>>
>>>>>> But this could be a quite quick fix (if easy to implement ?)
>>>>>>
>>>>>> I thought here on "streaming" and "raw content".
>>>>>>
>>>>>> - Streaming: I worked on a project where the http socket was 
>>>>>> directly streamed to the jdbc driver for file upload,  and the 
>>>>>> inverse for file download. For mail, upload is more smtp and is 
>>>>>> managed by jms queue (with  eventual claim-check pattern if 
>>>>>> needed). The download part is imap, and we load everything in 
>>>>>> memory before writing the response. We could imagine some kind of

>>>>>> listener that would be notified of each mail reading (reading 
>>>>>> done via stream). The listener would write the response via 
>>>>>> stream also, without storing it in memory.
>>>>>>
>>>>>> - "raw content". The store API forces to work with 
>>>>>> MailboxMembership that clearly isolates the headers from the 
>>>>>> content,... This is good to map the IMAP RFC that allows a client

>>>>>> to ask  for headers only for example. With the current API, we 
>>>>>> "impose" the store to structure the mails in content/header/... 
>>>>>> and if the store can not structure the mail (for example 
>>>>>> MailDir), we impose it to parse it, even if we have to return the

>>>>>> raw content in case of a classical plain FETCH command... I was 
>>>>>> wondering if we could not add a raw content storage obligation 
>>>>>> for all stores, and if we need to return the raw content, simply

>>>>>> use it. This could play nice with the stream option above.
>>>>>>
>>>>>> But yes, the batch approach is quicker to implement.
>>>>>>
>>>>>> Tks,
>>>>>>
>>>>>> Eric
>>>>>>
>>>>>>
>>>>>> On 11/10/2010 19:28, Norman Maurer wrote:
>>>>>>> Hi there,
>>>>>>>
>>>>>>> I tried to get my head around the cause of the OOM and I think
I 
>>>>>>> found
>>>>>>> a good solution. Let me outline the idea...
>>>>>>>
>>>>>>> The MessageManager interface expose many methods which expose

>>>>>>> kinds of
>>>>>>> Iterator<...>  objects. So the idea is to build up some
"batch 
>>>>>>> retrieve
>>>>>>> Iterator" which then retrieve the needed data in batches. So
the GC
>>>>>>> has a chance to kick in and free up resources. So here is an

>>>>>>> example..
>>>>>>>
>>>>>>> When call MessageManager.getMessages(...) (which returns
>>>>>>> Iterator<MessageResult>  ) and use a MessageRange of 1:1000
we 
>>>>>>> would
>>>>>>> fetch the MessagResult 1:100 as starting point. Then wait till
>>>>>>> Iterator.hasNext() or Iterator.next() is called and we have no
>>>>>>> MessageResult left we would fetch 101:200 and so on..
>>>>>>>
>>>>>>> I think this should work and will be much more efficient in 
>>>>>>> terms of
>>>>>>> memory. All this could get done in the "store" implementations
and
>>>>>>> could be configurable. Like use 100 as default batch count and

>>>>>>> be able
>>>>>>> to set it via a setter.
>>>>>>>
>>>>>>> WDYT ?
>>>>>>>
>>>>>>> Bye,
>>>>>>> Norman
>>>>>>>
>>>>>>>
>>>>>>> 2010/10/11 Norman Maurer<norman@apache.org>:
>>>>>>>> Hi Eric,
>>>>>>>>
>>>>>>>> Did you also have a look with wireshark what the exact command
and
>>>>>>>> argument was which triggered the OOM?
>>>>>>>>
>>>>>>>> Thx
>>>>>>>> Norman
>>>>>>>>
>>>>>>>> 2010/10/11, Eric Charles<eric@apache.org>:
>>>>>>>>> Hi Norman,
>>>>>>>>>
>>>>>>>>> There were 2 main problems:
>>>>>>>>> 1. The amq one which is now resolved tks to your last
commit
>>>>>>>>> 2. James no more responding on imap which is always caused
by 
>>>>>>>>> OOM (I
>>>>>>>>> missed some log the first time).
>>>>>>>>>
>>>>>>>>> For the second one, analysis of memory dump shows oom
comes 
>>>>>>>>> from huge
>>>>>>>>> usage of memory due to loading of message, headers,...
(in 
>>>>>>>>> case of
>>>>>>>>> 10.000 message fetch for example).
>>>>>>>>> I don't benefit from Lob streaming on derby database,
but it 
>>>>>>>>> won't help
>>>>>>>>> much because jpaheader for example also take much memory.
>>>>>>>>>
>>>>>>>>> Tks,
>>>>>>>>>
>>>>>>>>> Eric
>>>>>>>>>
>>>>>>>>> On 11/10/2010 13:10, Norman Maurer wrote:
>>>>>>>>>> Ok 4/5 is fixed now... Just to keep you updated..
>>>>>>>>>>
>>>>>>>>>> Bye.
>>>>>>>>>> Norman
>>>>>>>>>>
>>>>>>>>>> 2010/10/11 Norman Maurer<norman@apache.org>:
>>>>>>>>>>> Ok at least you can reproduce it, thats good
;) Did you take 
>>>>>>>>>>> a  thread
>>>>>>>>>>> dump ?
>>>>>>>>>>>
>>>>>>>>>>> Bye,
>>>>>>>>>>> Norman
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2010/10/11 Eric Charles<eric@apache.org>:
>>>>>>>>>>>> It's the same with latest thunderbird
>>>>>>>>>>>> I restarted disabling 'Check for new messages
on startup on 
>>>>>>>>>>>> all my
>>>>>>>>>>>> accounts.
>>>>>>>>>>>> If I go quickly from one folder to another,
I fall back in 
>>>>>>>>>>>> the endless
>>>>>>>>>>>> 'downloading'/'indexing'...
>>>>>>>>>>>> However, if I quietly click on 'Get Mail'
folder per 
>>>>>>>>>>>> folder, it's ok.
>>>>>>>>>>>>
>>>>>>>>>>>> I think we are still with Bug 1 (Bug 2 and
3 should be 
>>>>>>>>>>>> resolved if 1 is
>>>>>>>>>>>> resolved) for IMAP, fetching simultaneously
some folders.
>>>>>>>>>>>> Bug 4 is for amq.
>>>>>>>>>>>>
>>>>>>>>>>>> Tks,
>>>>>>>>>>>>
>>>>>>>>>>>> Eric
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 10/10/2010 20:03, Eric Charles wrote:
>>>>>>>>>>>>> I tried to resync thunderbird without
clicking on any folder.
>>>>>>>>>>>>> Still the same behaviour : "downloading
xxx on yyy", www 
>>>>>>>>>>>>> on zzz,...
>>>>>>>>>>>>>
>>>>>>>>>>>>> Wireshark tells me more: I never saw
such red/black lines 
>>>>>>>>>>>>> in the tcp
>>>>>>>>>>>>> stream (one red/black on every 5/10 tcp
packet: "segment 
>>>>>>>>>>>>> lost").
>>>>>>>>>>>>> 1783    8.626604    91.183.38.48    192.168.1.12
   
>>>>>>>>>>>>> IMAP    [TCP
>>>>>>>>>>>>> Previous
>>>>>>>>>>>>> segment lost] Response:
>>>>>>>>>>>>> ss.properties?rev=1005079&r1=1005078&r2=1005079&view=diff
>>>>>>>>>>>>>
>>>>>>>>>>>>> I was wondering if my cable was right:
>>>>>>>>>>>>> - tested plain http via cable: wireshark
is green.
>>>>>>>>>>>>> - tested thunderbird/james via wifi :
same black/red lines in
>>>>>>>>>>>>> wireshark.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have saved the dump and will analyze
further tomorrow, 
>>>>>>>>>>>>> but a tcp
>>>>>>>>>>>>> conversation selected from a "segment
lost" seems ok.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So for now (this may change), I think
we have:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. A client is in a stage that causes
the "segment lost" 
>>>>>>>>>>>>> tcp errors ==>
>>>>>>>>>>>>> Bug 1
>>>>>>>>>>>>> 2. Client/server conversation loops endless
==>    Bug 2
>>>>>>>>>>>>> 3.1. James finally hangs ==>    Bug
3
>>>>>>>>>>>>> 3.2. James finally gets oom ==>  
 Bug 3
>>>>>>>>>>>>> 4. Manual stop is needed.
>>>>>>>>>>>>> 5. After manual stop in state 3.1 or
3.2, there's a activemq
>>>>>>>>>>>>> java.io.EOFException: Chunk stream does
not exist at page: 
>>>>>>>>>>>>> 0 ==>    Bug 4
>>>>>>>>>>>>>
>>>>>>>>>>>>> So 4 bugs ?
>>>>>>>>>>>>> I will upgrade my thunderbird 3.0.3 on
linux to the latest 
>>>>>>>>>>>>> version and
>>>>>>>>>>>>> see
>>>>>>>>>>>>> if bug 1 is not resolved.
>>>>>>>>>>>>> Bug 4 may be resolved with 5.4.1 and
latest commits for 
>>>>>>>>>>>>> the james stop
>>>>>>>>>>>>> procedure.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Tks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Eric
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 10/10/2010 18:31, Eric Charles wrote:
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have on James 3 (trunk of 2 week
ago) my INBOX with 10 
>>>>>>>>>>>>>> subfolders,
>>>>>>>>>>>>>> some
>>>>>>>>>>>>>> of these subfolders having more than
10.000 mails.
>>>>>>>>>>>>>> I mainly use a PC, so the IMAP sync
is done regulary 
>>>>>>>>>>>>>> along the day.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I also have another PC I synchronize
once a week.
>>>>>>>>>>>>>> During the IMAP sync of that PC,
I selected randomly some 
>>>>>>>>>>>>>> subfolders
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> saw (this occured twice, but not
always...):
>>>>>>>>>>>>>> - Thunderbird syncs well during a
some minutes (10?)
>>>>>>>>>>>>>> - After, Thunderbird begins to say
"downloading xx of yy 
>>>>>>>>>>>>>> mails"..
>>>>>>>>>>>>>> .when
>>>>>>>>>>>>>> yy is reached, he says "downloading
ww of zz" where zz is 
>>>>>>>>>>>>>> a little
>>>>>>>>>>>>>> greater
>>>>>>>>>>>>>> than yy.
>>>>>>>>>>>>>> - I wait, wait, and finally have
timeout, and the mails 
>>>>>>>>>>>>>> are no more
>>>>>>>>>>>>>> viewable in thunderbird.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> James is stucked.
>>>>>>>>>>>>>> The first time I had OOM (I think),
today, I had no OOM, 
>>>>>>>>>>>>>> but James was
>>>>>>>>>>>>>> no
>>>>>>>>>>>>>> more reachable via IMAP, though accepting
mails via SMTP.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I stopped, and when restarting, I
had the following 
>>>>>>>>>>>>>> exception (James
>>>>>>>>>>>>>> was
>>>>>>>>>>>>>> not usable anymore):
>>>>>>>>>>>>>> INFO  18:16:37,646 | 
>>>>>>>>>>>>>> org.apache.activemq.store.kahadb.plist.PListStore
>>>>>>>>>>>>>> |
>>>>>>>>>>>>>> PListStore:activemq-data/localhost/tmp_storage
started
>>>>>>>>>>>>>> INFO  18:16:37,648 | 
>>>>>>>>>>>>>> org.apache.activemq.broker.BrokerService
| Using
>>>>>>>>>>>>>> Persistence Adapter:
>>>>>>>>>>>>>> KahaDBPersistenceAdapter[activemq-data/localhost/KahaDB]
>>>>>>>>>>>>>> INFO  18:16:38,248 | 
>>>>>>>>>>>>>> org.apache.activemq.store.kahadb.plist.PListStore
>>>>>>>>>>>>>> |
>>>>>>>>>>>>>> PListStore:../data/localhost/tmp_storage
started
>>>>>>>>>>>>>> ERROR 18:16:38,301 | 
>>>>>>>>>>>>>> org.apache.activemq.broker.BrokerService
| Failed
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> start ActiveMQ JMS Message Broker.
Reason: 
>>>>>>>>>>>>>> java.io.EOFException: Chunk
>>>>>>>>>>>>>> stream does not exist at page: 0
>>>>>>>>>>>>>> java.io.EOFException: Chunk stream
does not exist at page: 0
>>>>>>>>>>>>>>          at
>>>>>>>>>>>>>> org.apache.kahadb.page.Transaction$2.readPage(Transaction.java:454)

>>>>>>>>>>>>>>
>>>>>>>>>>>>>>          at
>>>>>>>>>>>>>> org.apache.kahadb.page.Transaction$2.<init>(Transaction.java:431)

>>>>>>>>>>>>>>
>>>>>>>>>>>>>>          at
>>>>>>>>>>>>>> org.apache.kahadb.page.Transaction.openInputStream(Transaction.java:428)

>>>>>>>>>>>>>>
>>>>>>>>>>>>>>          at
>>>>>>>>>>>>>> org.apache.kahadb.page.Transaction.load(Transaction.java:404)

>>>>>>>>>>>>>>
>>>>>>>>>>>>>>          at
>>>>>>>>>>>>>> org.apache.kahadb.page.Transaction.load(Transaction.java:361)

>>>>>>>>>>>>>>
>>>>>>>>>>>>>>          at
>>>>>>>>>>>>>> org.apache.activemq.store.kahadb.MessageDatabase$1.execute(MessageDatabase.java:243)

>>>>>>>>>>>>>>
>>>>>>>>>>>>>>          at
>>>>>>>>>>>>>> org.apache.kahadb.page.Transaction.execute(Transaction.java:728)

>>>>>>>>>>>>>>
>>>>>>>>>>>>>>          at
>>>>>>>>>>>>>> org.apache.activemq.store.kahadb.MessageDatabase.loadPageFile(MessageDatabase.java:230)

>>>>>>>>>>>>>>
>>>>>>>>>>>>>>          at
>>>>>>>>>>>>>> org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:309)

>>>>>>>>>>>>>>
>>>>>>>>>>>>>>          at
>>>>>>>>>>>>>> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:353)

>>>>>>>>>>>>>>
>>>>>>>>>>>>>>          at
>>>>>>>>>>>>>> org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:217)

>>>>>>>>>>>>>>
>>>>>>>>>>>>>>          at
>>>>>>>>>>>>>> org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:178)

>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sounds l ike 
>>>>>>>>>>>>>> https://issues.apache.org/activemq/browse/AMQ-2935.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To solve it, I had to remove the
activemq-data directory 
>>>>>>>>>>>>>> (btw, 2 weeks
>>>>>>>>>>>>>> ago was activemq 5.4.0 with 2 brokers
started and 
>>>>>>>>>>>>>> activemq-data in bin
>>>>>>>>>>>>>> directory).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I made a test to restart from scratch
my account in 
>>>>>>>>>>>>>> thunderbird, and
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> was OK.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is it because it does a incremental
sync and I select 
>>>>>>>>>>>>>> different
>>>>>>>>>>>>>> folders
>>>>>>>>>>>>>> (just to make things complicated
:) ) during the download ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Anyway, it is not easy to reproduce.
>>>>>>>>>>>>>> Activemq 5.4.1. may be worth to try,
but I'm not sure it 
>>>>>>>>>>>>>> the the
>>>>>>>>>>>>>> cause...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Tks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Eric
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To unsubscribe, e-mail: 
>>>>>>>>>>>>>> server-dev-unsubscribe@james.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail:

>>>>>>>>>>>>>> server-dev-help@james.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>>>>>>
>>>>>>>>>>>>> To unsubscribe, e-mail: 
>>>>>>>>>>>>> server-dev-unsubscribe@james.apache.org
>>>>>>>>>>>>> For additional commands, e-mail: 
>>>>>>>>>>>>> server-dev-help@james.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>>>>>
>>>>>>>>>>>> To unsubscribe, e-mail: 
>>>>>>>>>>>> server-dev-unsubscribe@james.apache.org
>>>>>>>>>>>> For additional commands, e-mail: 
>>>>>>>>>>>> server-dev-help@james.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>>>
>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>> For additional commands, e-mail: 
>>>>>>>>>> server-dev-help@james.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------

>>>>>>>>>
>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>> ---------------------------------------------------------------------

>>>>>>>
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------

>>>>>>
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message