james-server-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Norman Maurer <norman.mau...@googlemail.com>
Subject Re: OOM with Netty
Date Sat, 24 Apr 2010 13:07:30 GMT
Hi Eric,

thx for the report. Comments are inline...


2010/4/24 Eric Charles <eric.charles@u-mangate.com>:
> Hi All,
>
> I just got oom with netty impl.
> See http://apache.u-mangate.com/james/oom/20090424.png
> (http://apache.u-mangate.com/james/oom/index.html for history).
>
> Everything was fine with the trunk of last week that was really stable.
> I redeployed 3 days ago a new trunk version after a stop/start that showed
> ClassCastException of ImapMessage (ClassCastException was solved with the
> redeploy)

The ClassCastException should be fixed since yesterday. This was
caused by the EntityManager got confused by entities..


>
> Short infos about the today oom:
> - The logs had IOException "Bad file descriptor" (see copy of the exception
> after this mail)

This error seems to get thrown when you write to a closed stream...

> - The begining time of these exceptions in the log is 00:10, and there are
> numerous ones during 1 minute.
> - The heap dump automatically generated via -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/  occured on 00:10, exactly on the same time the
> IOException "Bad file descriptor" began.

Ok..

> - I forgot to lsof to see the open files... but when if the error came from
> too many open files, the received exception in the log would have been
> "java.io.IOException: too many open files".

right..

> - Seems like an infinite loop as Norman anticipated, but now stacktraces
> from logs seem to confirm this.
> - Looking at line 150 of
> http://svn.apache.org/viewvc/james/server/trunk/smtpserver/src/main/java/org/apache/james/smtpserver/DataLineJamesMessageHookHandler.java?view=markup
> , out.write(line) seem to be the place that triggers the exception, out
> variable being = (OutputStream)
> session.getState().get(SMTPConstants.DATA_MIMEMESSAGE_OUTPUTSTREAM);

Good catch will see how this could happen..

> - The server filesystem seems sane (new hardware, running with enough
> privilege, other applications running on the same machine without any
> problem). I don't see why I have a "Bad file descriptor"
>

See above..

> There are file in tmp with creation date that span 1 week (so not produced
> on 00:10):
> - A few messagestream*.ms
> - More rewindable*.tmp

Will investigate ..

>
> I restarted james.
> It still showed some issues (one account via pop3 gave errors in my client).
> oouch...
>
> I updated from trunk, redeployed, copied the var directory to migrate the
> account/domain/mail database, and restarted.
> Hopefully, now everything is working as before.
>
> So, we've got now a useful stacktrace that clearly shows where the oom
> "trigger" could be.
> I'm pretty sure that jconsole would have shown the same behaviour as
> http://apache.u-mangate.com/james/oom/oom.png
>
> Possible cause on the SMTP channel:
> - A weird behaviour of the smtp clients (hacking or respectfulness protocols
> clients)
> - A design in james causing infinite loop: thread clash (a thread accessing
> the same file as another thread?,...)
> - Additional James config (spamassassin or whatever) that leads to such
> behaviour
> - Hardware problem
>
> Currently, I run with default config + jdbcdomainlist.
> I except to receive another oom one of these days, unless the oom comes from
> an unactivated config (spamassassin or whatever).
>
> Tks,
>
> Eric
>
> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
> processing DATA.
> java.io.IOException: Bad file descriptor
>    at java.io.FileOutputStream.writeBytes(Native Method)
>    at java.io.FileOutputStream.write(FileOutputStream.java:247)
>    at
> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>    at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>    at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>    at
> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>    at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>    at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>    at
> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>    at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>    at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>    at
> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>    at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>    at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>    at
> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>    at
> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>    at
> org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:502)
>    at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
>    at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:119)
>    at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
>    at
> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:789)
>    at
> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:90)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:583)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:578)
>    at org.jboss.netty.channel.Channels.close(Channels.java:1065)
>    at
> org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:178)
>    at
> org.apache.james.smtpserver.netty.SMTPChannelUpstreamHandler.exceptionCaught(SMTPChannelUpstreamHandler.java:85)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:148)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>    at
> org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:619)
>    at
> org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:52)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:637)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:134)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
>    at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)
>    at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)
>    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)
>    at
> org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)
>    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)
>    at
> org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
>    at
> org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
>    at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>    at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>    at java.lang.Thread.run(Thread.java:619)
> ERROR 00:10:54,772 | james.smtpserver | Unknown error occurred while
> processing DATA.
> java.io.IOException: Bad file descriptor
>    at java.io.FileOutputStream.writeBytes(Native Method)
>    at java.io.FileOutputStream.write(FileOutputStream.java:247)
>    at
> org.apache.james.smtpserver.DataLineJamesMessageHookHandler.onLine(DataLineJamesMessageHookHandler.java:150)
>    at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>    at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>    at
> org.apache.james.protocols.smtp.core.ReceivedDataLineFilter.onLine(ReceivedDataLineFilter.java:54)
>    at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>    at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>    at
> org.apache.james.protocols.smtp.core.esmtp.MailSizeEsmtpExtension.onLine(MailSizeEsmtpExtension.java:151)
>    at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:75)
>    at
> org.apache.james.protocols.smtp.core.DataCmdHandler$DataLineFilterWrapper.onLine(DataCmdHandler.java:60)
>    at
> org.apache.james.socket.netty.LineHandlerUpstreamHandler.messageReceived(LineHandlerUpstreamHandler.java:62)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:237)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:117)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:391)
>    at
> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:317)
>    at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
>    at
> org.jboss.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:331)
>    at
> org.jboss.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:226)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
>    at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)
>    at
> org.jboss.netty.channel.SimpleChannelUpstreamHandler.channelDisconnected(SimpleChannelUpstreamHandler.java:191)
> ...
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>

Bye,
Norman

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org


Mime
View raw message