qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Rudyy (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (QPID-6710) NPE masks IOException on running out of disk space
Date Mon, 31 Aug 2015 15:09:45 GMT

     [ https://issues.apache.org/jira/browse/QPID-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alex Rudyy updated QPID-6710:
-----------------------------
    Attachment:     (was: delegate-exception-throwing-to-je-and-handle-je-exceptions-everywhere.diff)

> NPE masks IOException on running out of disk space
> --------------------------------------------------
>
>                 Key: QPID-6710
>                 URL: https://issues.apache.org/jira/browse/QPID-6710
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Broker
>            Reporter: Lorenz Quack
>         Attachments: 0001-QPID-6710-Handle-JE-exceptions-in-order-to-restart-e.patch,
0001-QPID-6710-Java-Broker-NPE-masks-IOException-on-runni.patch
>
>
> When the broker is running out of disk space the following issue can occur.
> {noformat}
> ########################################################################
> #
> # Unhandled Exception org.apache.qpid.server.store.StoreException: Unexpected exception
occurred on store operation in Thread virtualhost-default-iopool-55
> #
> # Exiting
> #
> ########################################################################
> org.apache.qpid.server.store.StoreException: Unexpected exception occurred on store operation
>         at org.apache.qpid.server.store.berkeleydb.StandardEnvironmentFacade.handleDatabaseException(StandardEnvironmentFacade.java:287)
>         at org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.removeMessage(AbstractBDBMessageStore.java:317)
>         at org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage.remove(AbstractBDBMessageStore.java:1234)
>         at org.apache.qpid.server.message.AbstractServerMessageImpl.decrementReference(AbstractServerMessageImpl.java:101)
>         at org.apache.qpid.server.message.AbstractServerMessageImpl.access$500(AbstractServerMessageImpl.java:37)
>         at org.apache.qpid.server.message.AbstractServerMessageImpl$Reference.release(AbstractServerMessageImpl.java:275)
>         at org.apache.qpid.server.protocol.v0_8.AMQChannel.deliverCurrentMessageIfComplete(AMQChannel.java:526)
>         at org.apache.qpid.server.protocol.v0_8.AMQChannel.publishContentBody(AMQChannel.java:655)
>         at org.apache.qpid.server.protocol.v0_8.AMQChannel.receiveMessageContent(AMQChannel.java:2519)
>         at org.apache.qpid.framing.ContentBody.process(ContentBody.java:105)
>         at org.apache.qpid.codec.AMQDecoder.processFrame(AMQDecoder.java:394)
>         at org.apache.qpid.server.protocol.v0_8.BrokerDecoder.doProcessFrame(BrokerDecoder.java:114)
>         at org.apache.qpid.server.protocol.v0_8.BrokerDecoder.access$000(BrokerDecoder.java:37)
>         at org.apache.qpid.server.protocol.v0_8.BrokerDecoder$1.run(BrokerDecoder.java:80)
>         at org.apache.qpid.server.protocol.v0_8.BrokerDecoder$1.run(BrokerDecoder.java:76)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.qpid.server.protocol.v0_8.BrokerDecoder.processFrame(BrokerDecoder.java:75)
>         at org.apache.qpid.codec.AMQDecoder.processInput(AMQDecoder.java:370)
>         at org.apache.qpid.codec.AMQDecoder.decodeBuffer(AMQDecoder.java:259)
>         at org.apache.qpid.server.protocol.v0_8.AMQPConnection_0_8$2.run(AMQPConnection_0_8.java:319)
>         at org.apache.qpid.server.protocol.v0_8.AMQPConnection_0_8$2.run(AMQPConnection_0_8.java:299)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:356)
>         at org.apache.qpid.server.protocol.v0_8.AMQPConnection_0_8.received(AMQPConnection_0_8.java:298)
>         at org.apache.qpid.server.transport.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:138)
>         at org.apache.qpid.server.transport.NonBlockingConnection.processAmqpData(NonBlockingConnection.java:465)
>         at org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:45)
>         at org.apache.qpid.server.transport.NonBlockingConnection.processData(NonBlockingConnection.java:398)
>         at org.apache.qpid.server.transport.NonBlockingConnection.readAndProcessData(NonBlockingConnection.java:349)
>         at org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.doRead(NonBlockingConnectionPlainDelegate.java:39)
>         at org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:337)
>         at org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:229)
>         at org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:119)
>         at org.apache.qpid.server.transport.NetworkConnectionScheduler.access$000(NetworkConnectionScheduler.java:37)
>         at org.apache.qpid.server.transport.NetworkConnectionScheduler$2.run(NetworkConnectionScheduler.java:100)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>         at org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.removeMessage(AbstractBDBMessageStore.java:227)
>         ... 37 more
> {noformat}
> Broker log file might have the issue reported in some cases as:
> {noformat}
> 2015-07-23 05:19:30,460 INFO  [default:FileSystemSpaceChecker] (q.m.v.filesystem_full)
- [default:FileSystemSpaceChecker] [vh(/default)/ms(BDBMessageStore)] VHT-1006 : Filesystem
is over 90 per cent full, enforcing flow control.
> 2015-07-23 05:19:46,668 ERROR [IO-/169.124.113.161:58633] (o.a.q.s.u.ServerScopedRuntimeException)
- Unexpected exception occurred on store operation
> com.sleepycat.je.LogWriteException: (JE 5.0.104) Environment must be closed, caused by:
com.sleepycat.je.LogWriteException: Environment invalid because of previous exception: (JE
5.0.104) /apps/qpid/destruction-tests/qpid-broker/6.0.0-SNAPSHOT/work/default/messages java.io.IOException:
No space left on device LOG_WRITE: IOExcep
> {noformat}
> However, in some other cases exception is not get stored in the logs leaving only NPE
(reported into system output) to be the only evidence of an issue which might confuse the
user about the nature of a problem



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Mime
View raw message