james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Norman Maurer <nor...@apache.org>
Subject Re: OOM with Netty
Date Sat, 24 Apr 2010 13:13:28 GMT
Could you upload the heap dump too ?


2010/4/24, Norman Maurer <norman.maurer@googlemail.com>:
> Hi Eric,
>
> thx for the report. Comments are inline...
>
>
> 2010/4/24 Eric Charles <eric.charles@u-mangate.com>:
>> Hi All,
>>
>> I just got oom with netty impl.
>> See http://apache.u-mangate.com/james/oom/20090424.png
>> (http://apache.u-mangate.com/james/oom/index.html for history).
>>
>> Everything was fine with the trunk of last week that was really stable.
>> I redeployed 3 days ago a new trunk version after a stop/start that
>> showed
>> ClassCastException of ImapMessage (ClassCastException was solved with the
>> redeploy)
>
> The ClassCastException should be fixed since yesterday. This was
> caused by the EntityManager got confused by entities..
>
>
>>
>> Short infos about the today oom:
>> - The logs had IOException "Bad file descriptor" (see copy of the
>> exception
>> after this mail)
>
> This error seems to get thrown when you write to a closed stream...
>
>> - The begining time of these exceptions in the log is 00:10, and there
>> are
>> numerous ones during 1 minute.
>> - The heap dump automatically generated via
>> -XX:+HeapDumpOnOutOfMemoryError
>> -XX:HeapDumpPath=/  occured on 00:10, exactly on the same time the
>> IOException "Bad file descriptor" began.
>
> Ok..
>
>> - I forgot to lsof to see the open files... but when if the error came
>> from
>> too many open files, the received exception in the log would have been
>> "java.io.IOException: too many open files".
>
> right..
>
>> - Seems like an infinite loop as Norman anticipated, but now stacktraces
>> from logs seem to confirm this.
>> - Looking at line 150 of
>> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=markup
>> , out.write(line) seem to be the place that triggers the exception, out
>> variable being = (OutputStream)
>> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);
>
> Good catch will see how this could happen..
>
>> - The server filesystem seems sane (new hardware, running with enough
>> privilege, other applications running on the same machine without any
>> problem). I don't see why I have a "Bad file descriptor"
>>
>
> See above..
>
>> There are file in tmp with creation date that span 1 week (so not
>> produced
>> on 00:10):
>> - A few messagestream*.ms
>> - More rewindable*.tmp
>
> Will investigate ..
>
>>
>> I restarted james.
>> It still showed some issues (one account via pop3 gave errors in my
>> client).
>> oouch...
>>
>> I updated from trunk, redeployed, copied the var directory to migrate the
>> account/domain/mail database, and restarted.
>> Hopefully, now everything is working as before.
>>
>> So, we've got now a useful stacktrace that clearly shows where the oom
>> "trigger" could be.
>> I'm pretty sure that jconsole would have shown the same behaviour as
>> http://apache.u-mangate.com/james/oom/oom.png
>>
>> Possible cause on the SMTP channel:
>> - A weird behaviour of the smtp clients (hacking or respectfulness
>> protocols
>> clients)
>> - A design in james causing infinite loop: thread clash (a thread
>> accessing
>> the same file as another thread?,...)
>> - Additional James config (spamassassin or whatever) that leads to such
>> behaviour
>> - Hardware problem
>>
>> Currently, I run with default config + jdbcdomainlist.
>> I except to receive another oom one of these days, unless the oom comes
>> from
>> an unactivated config (spamassassin or whatever).
>>
>> Tks,
>>
>> Eric
>>
>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>> processing DATA.
>> java.io.IOException: Bad file descriptor
>>    at java.io.FileOutputStream.writeBytes(Native Method)
>>    at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>    at
>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>    at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>    at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>    at
>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>    at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>    at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>    at
>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>    at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>    at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>    at
>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>    at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>    at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>    at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>    at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>    at
>> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
>>    at
>> org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
>>    at
>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
>>    at
>> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
>>    at
>> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
>>    at
>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
>>    at org.jboss.netty.channel.Channels.close(Channels.java:1065)
>>    at
>> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
>>    at
>> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>    at
>> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
>>    at
>> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>>    at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
>>    at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
>>    at
>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>>    at
>> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>>    at
>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>>    at
>> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>>    at
>> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>>    at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>    at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>    at java.lang.Thread.run(Thread.java:619)
>> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
>> processing DATA.
>> java.io.IOException: Bad file descriptor
>>    at java.io.FileOutputStream.writeBytes(Native Method)
>>    at java.io.FileOutputStream.write(FileOutputStream.java:247)
>>    at
>> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>>    at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>    at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>    at
>> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>>    at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>    at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>    at
>> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>>    at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>>    at
>> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>>    at
>> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>>    at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>>    at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>>    at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>>    at
>> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>>    at
>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>>    at
>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>> ...
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
> Bye,
> Norman
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message