james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Norman Maurer <norman.mau...@googlemail.com>
Subject Re: OutOfMemory exception
Date Thu, 01 Apr 2010 07:52:53 GMT
After reading the page about prefetch limits again, you should maybe
set the prefetchlimit to 1 too..

http://activemq.apache.org/what-is-the-prefetch-limit-for.html

Pooled Connections and prefetch

Consuming messages from a connection pool can be problematic due to
prefetch. Unconsumed prefetched messages are only released when a
connection is closed, but with a pooled connection the connection
close is deferred (for reuse) till the connection pool closes. This
leaves prefetched messages unconsumed till the connection is reused.
This feature can present as missing or out-of-sequence messages when
there is more than one connection in the pool.
One solution is to use pooled connections for producers and a
non-pooled connection for consumers. This might have performance
impacts on the consumer side, if multiple threads try to consume
messages at a fast rate. Alternatively, reduce the pool size to 1 for
consumers. A third alternative is to reduce the prefetchSize to 1 or 0
with the pooled connection factory. When using Spring JMS and
MessageDrivenPojo, you cannot use a prefetch of 0, so use 1 instead.



As we use a connection  pool...

Bye,
Norman


2010/4/1 Norman Maurer <norman.maurer@googlemail.com>:
> Hi Eric,
>
> could you try if this helps...
>
> In spring-beans.xml replace the jmsConnectionFactory block with:
>
>    <!-- jms connection pooling  -->
>    <bean id="jmsConnectionFactory" class="org.apache.activemq.pool.PooledConnec
> tionFactory" destroy-method="stop">
>      <property name="connectionFactory">
>        <bean class="org.apache.activemq.ActiveMQConnectionFactory">
>          <property name="brokerURL">
>            <value>vm://localhost?broker.useJmx=false&amp;jms.prefetchPolicy.all
> =50</value>
>          </property>
>        </bean>
>      </property>
>    </bean>
>
> Bye,
> Norman
>
> 2010/4/1 Eric Charles <eric.charles@u-mangate.com>:
>> 2) Launching james, memory usage is 300MB. This seems normal and holding the
>> traffic well. At a certain time, memory jumps to 3GB (I gave much memory via
>> -Xmx).
>> Maybe a certain condition produces this.
>> Debugging in eclipse shows me that many threads are created by camel. I also
>> had the impression that some threads were falling and other ones were
>> recreated. If this is the case, is it normal.
>> I'm going to look at activemq/camel (I don't know them really), and also
>> still read the http://activemq.apache.org/javalangoutofmemory.html,
>> http://activemq.apache.org/my-producer-blocks.html,...
>>
>> I have the impression that everything is working fine, but at a certain
>> moment, there is a condition that makes the memory usage rise extensively
>> (not a linear growth, but really a stair).
>>
>> Bye,
>> Eric
>>
>>
>> On 04/01/2010 09:24 AM, Norman Maurer wrote:
>>>
>>> Hi Eric,
>>>
>>> thx again for your tests...
>>>
>>> So I think we have two independent problems here:
>>>
>>> 1) The .m64 files which are not deleted always:
>>>
>>> My guess here is that there is something wrong with processing Mail
>>> objects with state Null. I will investigate futher.. I also found a
>>> possible problem with .m64 files which get not deleted when a
>>> Exception is throws while processing the camel routes. I will patch
>>> this soon.
>>>
>>> 2) The OOM
>>>
>>> My guess here is that ActiveMQ is not freeing up memory. Not sure why yet
>>>
>>> I think 2) is the more criticial problem and should get higher
>>> priority to get fixed..
>>>
>>> Bye,
>>> Norman
>>>
>>>
>>>
>>> 2010/4/1 Eric Charles<eric.charles@u-mangate.com>:
>>>
>>>>
>>>> Hi Norman,
>>>>
>>>> Tks for the smpt-source pointer.
>>>> I looked at it, but finally, I think I will develop those little classes
>>>> to
>>>> have more control on it.
>>>> I will put it on github.
>>>>
>>>> I launched latest trunk james yesterday evening.
>>>> During the night, 3 hours after the launch, the following occured:
>>>>
>>>> KahDB slower and slower...
>>>>
>>>> INFO  23:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>> Slow
>>>> KahaDB access: cleanup took 2330
>>>> INFO  23:38:04,591 | james.mailetcontext | Error while storing mail.
>>>> org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be
>>>> parsed.;
>>>>  nested exception is:
>>>>    org.apache.james.imap.mailbox.StorageException: failed. Transaction
>>>> commit failed.;
>>>>  nested exception is:
>>>> <openjpa-1.2.1-r752877:753278 fatal store error>
>>>> org.apache.openjpa.persistence.RollbackException: The transaction has
>>>> been
>>>> rolled back.  See the nested exceptions for details on the errors that
>>>> occurred.
>>>>    at
>>>>
>>>> org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265)
>>>>
>>>>    at
>>>>
>>>> org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:130)
>>>>    ... 55 more
>>>> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
>>>> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
>>>> 1363215207 INSERT INTO Message (id, bodyStartOctet, content,
>>>> contentOctets,
>>>> mediaType, subType, textu
>>>> alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=(long) 2678, (int)
>>>> 1260,
>>>> (byte[]) [B@7a00e70c, (long) 47887, (String) text, (String) html, (long)
>>>> 876]} [code=0, state=XJ001]
>>>>    at
>>>>
>>>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192)
>>>>    at
>>>>
>>>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57)
>>>>    at
>>>>
>>>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866)
>>>>    at
>>>>
>>>> org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269)
>>>>
>>>> So still a OOM exception that was shown by yet-another-component (in this
>>>> case, the StoreMailbox).
>>>>
>>>>
>>>> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
>>>> job).
>>>> All 4 files were 0 bytes.
>>>>
>>>> I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB
>>>> max
>>>> memory).
>>>>
>>>> With the previous parameters ( -Xmx512m), the process was taking the
>>>> whole
>>>> 512MB.
>>>>
>>>> Tks,
>>>> Eric
>>>>
>>>>
>>>> On 03/31/2010 08:26 PM, Norman Maurer wrote:
>>>>
>>>>>
>>>>> Hi eric,
>>>>>
>>>>> thx for all your help, maybe smtp-source ( which is included in
>>>>> postfix ) can help you with stress testing.
>>>>>
>>>>> Bye
>>>>> Norman
>>>>>
>>>>> 2010/3/31, Eric Charles<eric.charles@u-mangate.com>:
>>>>>
>>>>>
>>>>>>
>>>>>> Hi Norman,
>>>>>>
>>>>>> There are not so much .m64 files, so even it it shows that a mail
>>>>>> processing didn't reach the end and may have cause a leak, I don't
>>>>>> think
>>>>>> it would make crash my process so quick (between 10 and 24 hours).
>>>>>>
>>>>>> I quickly analysed a few dumps last week : there is always a class
that
>>>>>> took 30/40% of the memory. Sometimes a activemq class, sometimes
not
>>>>>> (don't remember the details).
>>>>>> Also notable, the stack trace (the place where the OOM gives problems)
>>>>>> varies.
>>>>>> Today stack was:
>>>>>> org.apache.camel.RuntimeCamelException:
>>>>>> org.apache.camel.CamelExchangeException: Error processing Exchange.
>>>>>> Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102,
>>>>>> responseRequired = true, messageId =
>>>>>> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination =
null,
>>>>>> originalTransactionId = null, producerId =
>>>>>> ID:srv001-51032-1270011735798-2:0:24:1, destination =
>>>>>> queue://processor.root, transactionId = null, expiration = 0, timestamp
>>>>>> = 1270014325685, arrival = 0, brokerInTime = 1270014326893,
>>>>>> brokerOutTime = 1270014333023, correlationId = null, replyTo = null,
>>>>>> persistent = true, type = null, priority = 4, groupID = null,
>>>>>> groupSequence = 0, targetConsumerId = null, compressed = false, userID
>>>>>> =
>>>>>> null, content = org.apache.activemq.util.ByteSequence@69b568d0,
>>>>>> marshalledProperties = null, dataStructure = null, redeliveryCounter
=
>>>>>> 0, size = 10718, properties = null, readOnlyProperties = true,
>>>>>> readOnlyBody = true, droppable = false}]. Caused by:
>>>>>> [java.lang.OutOfMemoryError - Java heap space]
>>>>>>      at
>>>>>>
>>>>>>
>>>>>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055)
>>>>>>      at
>>>>>>
>>>>>>
>>>>>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154)
>>>>>> ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> I also think the tmp files are not the main cause.
>>>>>>
>>>>>> I have now james running in eclipse and I made already made a little
>>>>>> trip in the code. Quite impressive since last time I looked at it.
>>>>>> I will try to stress james with a small smtp/pop3 client (I used
>>>>>> postage, but a simple class with commons-net behind) may be easier
and
>>>>>> see what happens (eventually with eclipse profiler).
>>>>>>
>>>>>> It will be for this weekend for me.
>>>>>>
>>>>>> Bye,
>>>>>>
>>>>>> Eric
>>>>>>
>>>>>>
>>>>>> On 03/31/2010 07:11 PM, Norman Maurer wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Hi Eric,
>>>>>>>
>>>>>>> thx for keeping us in the loop... I'm still not sure why the
.m64
>>>>>>> files are still in the tmp folder sometimes.. But I suspect the
files
>>>>>>> are not the cause of the OOM.
>>>>>>>
>>>>>>> I didn't get a OOM since yesterday morning (the time I deployed
>>>>>>> current trunk version), but just found to new .m64 files..
>>>>>>>
>>>>>>> So I'm still searching for the real cause. If nothing helps I
will
>>>>>>> need to use a profiler to find th leak.
>>>>>>>
>>>>>>> Bye,
>>>>>>> Norman
>>>>>>>
>>>>>>> 2010/3/31 Eric Charles<eric.charles@u-mangate.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Hi Norman,
>>>>>>>>
>>>>>>>> I had defined the Null mailet
>>>>>>>>
>>>>>>>> <mailet match="HostIsLocal" class="Null">
>>>>>>>> <processor>     local-address-error</processor>
>>>>>>>> <notice>550 - Requested action not taken: no such user
here</notice>
>>>>>>>> </mailet>
>>>>>>>>
>>>>>>>> but now, I use the standard config
>>>>>>>>
>>>>>>>> <mailet match="HostIsLocal" class="ToProcessor">
>>>>>>>> <processor>     local-address-error</processor>
>>>>>>>> <notice>550 - Requested action not taken: no such user
here</notice>
>>>>>>>> </mailet>
>>>>>>>>
>>>>>>>> I still have the OOM.
>>>>>>>>
>>>>>>>> I will now deploy a fresh svn with your last commits and
enable the
>>>>>>>> ValidRcptHandler.
>>>>>>>> I keep you posted with the result,
>>>>>>>>
>>>>>>>> Tks,
>>>>>>>> Eric
>>>>>>>>
>>>>>>>>
>>>>>>>> On 03/30/2010 06:49 AM, Norman Maurer wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Eric,
>>>>>>>>>
>>>>>>>>> you said all the files are related to address-errors
, could you
>>>>>>>>> show
>>>>>>>>> me your address error processor config?
>>>>>>>>> Does the Problem still exist when you enable the ValidRcptHandler
in
>>>>>>>>> the smtpserver.xml file?
>>>>>>>>>
>>>>>>>>> Thx
>>>>>>>>> Norman
>>>>>>>>>
>>>>>>>>> 2010/3/30, Eric Charles<eric.charles@u-mangate.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Oops, no, the files are still there (only unknown@known.com).
>>>>>>>>>> Eric
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 03/29/2010 10:16 PM, Eric Charles wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Norman,
>>>>>>>>>>>
>>>>>>>>>>> I just deployed your new commit with the new
camel DisposeProcess.
>>>>>>>>>>> Good news : I don't see anymore the m64 file
in tmp (well I see 1
>>>>>>>>>>> file, on the second after, it is no more there,
so the dispose
>>>>>>>>>>> works
>>>>>>>>>>> as it should).
>>>>>>>>>>>
>>>>>>>>>>> I keep you posted with our eventual future OOM.
>>>>>>>>>>>
>>>>>>>>>>> Tks,
>>>>>>>>>>> Eric
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Norman,
>>>>>>>>>>>>
>>>>>>>>>>>> The .m64 are all to "unkown user" but to
"well known domain" (so
>>>>>>>>>>>> for
>>>>>>>>>>>> <unknown@known.com>).
>>>>>>>>>>>> They are from various size (with and without
attachment).
>>>>>>>>>>>> They are well formed (I can open the downloaded
file with
>>>>>>>>>>>> thunderbird)
>>>>>>>>>>>>
>>>>>>>>>>>> However, when I sent a mail to unknown@known.com,
I don't see it
>>>>>>>>>>>> in
>>>>>>>>>>>> the /tmp
>>>>>>>>>>>>
>>>>>>>>>>>> Tks,
>>>>>>>>>>>> Eric
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>
>>>>>>>>>>>>> sure.. we have to find the OOM cause.
What would be interesting,
>>>>>>>>>>>>> could
>>>>>>>>>>>>> you check somehow if the .m64 files are
files which are related
>>>>>>>>>>>>> to
>>>>>>>>>>>>> successfully delivered mail ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2010/3/29 Eric Charles<eric.charles@u-mangate.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Norman,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Done. Now, we have to wait...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I saw some created *.m64 that were
removed.
>>>>>>>>>>>>>> But there are other ones that remains
in /tmp.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I sometimes run a jmap (java memory
map) to produce a heap dump
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> analyse
>>>>>>>>>>>>>> it.
>>>>>>>>>>>>>> At the beginning, everything seems
ok, and often, when I come
>>>>>>>>>>>>>> back,
>>>>>>>>>>>>>> the OOM
>>>>>>>>>>>>>> has already occured.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I changed the -Xmx512m to -Xmx256m
to have a quicker exception
>>>>>>>>>>>>>> (if
>>>>>>>>>>>>>> any,
>>>>>>>>>>>>>> let's hope not).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The *.m64 are a clue and we should
ensure that they are removed
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> all
>>>>>>>>>>>>>> circumstances.
>>>>>>>>>>>>>> This may be the easy part, as we
have some visible clues (the
>>>>>>>>>>>>>> files).
>>>>>>>>>>>>>> We should also ensure after that
there are no other causes to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> leaks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Tks,
>>>>>>>>>>>>>> Eric
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer
wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I found the cause for the not
deleted temporary files.
>>>>>>>>>>>>>>> Hopefully
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> is the cause of the OOM. Could
try to svn up and run again ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2010/3/29 Norman Maurer<norman.maurer@googlemail.com>:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> thx for the report. I see
exact the same problem today here..
>>>>>>>>>>>>>>>> (The
>>>>>>>>>>>>>>>> OOM). Didn't notice the files
in the tmp folder, but I think
>>>>>>>>>>>>>>>> thats
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> good pointer. I will try
to debug the problem later or
>>>>>>>>>>>>>>>> tomorrow.
>>>>>>>>>>>>>>>> But I
>>>>>>>>>>>>>>>> suspect you are right about
the tmp files and jms producers..
>>>>>>>>>>>>>>>> Did
>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>> run a kill -3 pid to see
what threads are active etc ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ps: Patches are welcome :)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2010/3/29 Eric Charles<eric.charles@u-mangate.com>:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The last two weeks, I
deployed various trunk snapshots.
>>>>>>>>>>>>>>>>> After james running less
than 1 day, I always ran into a
>>>>>>>>>>>>>>>>> OutOfMemory
>>>>>>>>>>>>>>>>> Exception.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I analysed the logs (for
example STACK TRACE 1 in annex),
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>> Camel complaining when
trying to deliver on JMS queue.
>>>>>>>>>>>>>>>>> I tried to adapt the
ActiveMQ configuration based on
>>>>>>>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html
but
>>>>>>>>>>>>>>>>> nothing
>>>>>>>>>>>>>>>>> helped.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I also analyzed various
heap dump :sometimes, 65% was used
>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession,
>>>>>>>>>>>>>>>>> sometimes
>>>>>>>>>>>>>>>>> by a
>>>>>>>>>>>>>>>>> activemq class,...
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message