james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Charles <eric.char...@u-mangate.com>
Subject Re: OOM with Netty
Date Sat, 24 Apr 2010 13:43:42 GMT
Hi Norman,

You can download it from 
http://apache.u-mangate.com/james/oom/java_pid23898.hprof.zip


"This error seems to get thrown when you write to a closed stream.."
Could it be a thread-safety problem ? (on thread using the session/state 
of another one ?)

I will also monitor with lsof -p <pid> | grep deleted | wc -l

As config change, I activated a few days ago the ValidRcptHandler 
fastfail handler.

Tks,

Eric


On 04/24/2010 03:13 PM, Norman Maurer wrote:
> Could you upload the heap dump too ?
>
>
> 2010/4/24, Norman Maurer<norman.maurer@googlemail.com>:
>    
>> Hi Eric,
>>
>> thx for the report. Comments are inline...
>>
>>
>> 2010/4/24 Eric Charles<eric.charles@u-mangate.com>:
>>      
>>> Hi All,
>>>
>>> I just got oom with netty impl.
>>> See http://apache.u-mangate.com/james/oom/20090424.png
>>> (http://apache.u-mangate.com/james/oom/index.html for history).
>>>
>>> Everything was fine with the trunk of last week that was really stable.
>>> I redeployed 3 days ago a new trunk version after a stop/start that
>>> showed
>>> ClassCastException of ImapMessage (ClassCastException was solved with the
>>> redeploy)
>>>        
>> The ClassCastException should be fixed since yesterday. This was
>> caused by the EntityManager got confused by entities..
>>
>>
>>      
>>> Short infos about the today oom:
>>> - The logs had IOException "Bad file descriptor" (see copy of the
>>> exception
>>> after this mail)
>>>        
>> This error seems to get thrown when you write to a closed stream...
>>
>>      
>>> - The begining time of these exceptions in the log is 00:10, and there
>>> are
>>> numerous ones during 1 minute.
>>> - The heap dump automatically generated via
>>> -XX:+HeapDumpOnOutOfMemoryError
>>> -XX:HeapDumpPath=/  occured on 00:10, exactly on the same time the
>>> IOException "Bad file descriptor" began.
>>>        
>> Ok..
>>
>>      
>>> - I forgot to lsof to see the open files... but when if the error came
>>> from
>>> too many open files, the received exception in the log would have been
>>> "java.io.IOException: too many open files".
>>>        
>> right..
>>
>>      
>>> - Seems like an infinite loop as Norman anticipated, but now stacktraces
>>> from logs seem to confirm this.
>>> - Looking at line 150 of
>>> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=markup
>>> , out.write(line) seem to be the place that triggers the exception, out
>>> variable being = (OutputStream)
>>> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
>>>        
>> Good catch will see how this could happen..
>>
>>      
>>> - The server filesystem seems sane (new hardware, running with enough
>>> privilege, other applications running on the same machine without any
>>> problem). I don't see why I have a "Bad file descriptor"
>>>
>>>        
>> See above..
>>
>>      
>>> There are file in tmp with creation date that span 1 week (so not
>>> produced
>>> on 00:10):
>>> - A few messagestream*.ms
>>> - More rewindable*.tmp
>>>        
>> Will investigate ..
>>
>>      
>>> I restarted james.
>>> It still showed some issues (one account via pop3 gave errors in my
>>> client).
>>> oouch...
>>>
>>> I updated from trunk, redeployed, copied the var directory to migrate the
>>> account/domain/mail database, and restarted.
>>> Hopefully, now everything is working as before.
>>>
>>> So, we've got now a useful stacktrace that clearly shows where the oom
>>> "trigger" could be.
>>> I'm pretty sure that jconsole would have shown the same behaviour as
>>> http://apache.u-mangate.com/james/oom/oom.png
>>>
>>> Possible cause on the SMTP channel:
>>> - A weird behaviour of the smtp clients (hacking or respectfulness
>>> protocols
>>> clients)
>>> - A design in james causing infinite loop: thread clash (a thread
>>> accessing
>>> the same file as another thread?,...)
>>> - Additional James config (spamassassin or whatever) that leads to such
>>> behaviour
>>> - Hardware problem
>>>
>>> Currently, I run with default config + jdbcdomainlist.
>>> I except to receive another oom one of these days, unless the oom comes
>>> from
>>> an unactivated config (spamassassin or whatever).
>>>
>>> Tks,
>>>
>>> Eric
>>>
>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>> processing DATA.
>>> java.io.IOException: Bad file descriptor
>>>     at java.io.FileOutputStream.writeBytes(Native Method)
>>>     at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>     at
>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>     at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>     at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>     at
>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>     at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>     at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>     at
>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>     at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>     at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>     at
>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>     at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>     at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>     at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>     at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>     at
>>> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
>>>     at
>>> org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
>>>     at
>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
>>>     at
>>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
>>>     at
>>> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
>>>     at
>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
>>>     at org.jboss.netty.channel.Channels.close(Channels.java:1065)
>>>     at
>>> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
>>>     at
>>> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>     at
>>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
>>>     at
>>> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>>     at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
>>>     at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
>>>     at
>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>>>     at
>>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>>>     at
>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>>>     at
>>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>>     at
>>> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>>>     at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>     at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>     at java.lang.Thread.run(Thread.java:619)
>>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>>> processing DATA.
>>> java.io.IOException: Bad file descriptor
>>>     at java.io.FileOutputStream.writeBytes(Native Method)
>>>     at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>>     at
>>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>>     at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>     at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>     at
>>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>>     at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>     at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>     at
>>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>>     at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>>     at
>>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>>     at
>>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>>     at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>>     at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>>     at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>>     at
>>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>>     at
>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>>     at
>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>> ...
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>        
>> Bye,
>> Norman
>>
>>      
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>    


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message