qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (QPID-6710) NPE masks IOException on running out of disk space
Date Mon, 31 Aug 2015 17:21:46 GMT

    [ https://issues.apache.org/jira/browse/QPID-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723721#comment-14723721
] 

ASF subversion and git services commented on QPID-6710:
-------------------------------------------------------

Commit 1700280 from orudyy@apache.org in branch 'java/trunk'
[ https://svn.apache.org/r1700280 ]

QPID-6710: [Java Broker] Rely on JE exceptions thrown for invalid environment and handle them
in order to restart environment as early as possible

           ( work by Lorenz Quack <quack.lorenz@gmail.com> and Alex Rudyy <orudyy@gmail.com>)

> NPE masks IOException on running out of disk space
> --------------------------------------------------
>
>                 Key: QPID-6710
>                 URL: https://issues.apache.org/jira/browse/QPID-6710
>             Project: Qpid
>          Issue Type: Bug
>          Components: Java Broker
>            Reporter: Lorenz Quack
>         Attachments: 0001-QPID-6710-Handle-JE-exceptions-in-order-to-restart-e.patch,
0001-QPID-6710-Java-Broker-NPE-masks-IOException-on-runni.patch
>
>
> When the broker is running out of disk space the following issue can occur.
> {noformat}
> ########################################################################
> #
> # Unhandled Exception org.apache.qpid.server.store.StoreException: Unexpected exception
occurred on store operation in Thread virtualhost-default-iopool-55
> #
> # Exiting
> #
> ########################################################################
> org.apache.qpid.server.store.StoreException: Unexpected exception occurred on store operation
>         at org.apache.qpid.server.store.berkeleydb.StandardEnvironmentFacade.handleDatabaseException(StandardEnvironmentFacade.java:287)
>         at org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.removeMessage(AbstractBDBMessageStore.java:317)
>         at org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage.remove(AbstractBDBMessageStore.java:1234)
>         at org.apache.qpid.server.message.AbstractServerMessageImpl.decrementReference(AbstractServerMessageImpl.java:101)
>         at org.apache.qpid.server.message.AbstractServerMessageImpl.access$500(AbstractServerMessageImpl.java:37)
>         at org.apache.qpid.server.message.AbstractServerMessageImpl$Reference.release(AbstractServerMessageImpl.java:275)
>         at org.apache.qpid.server.protocol.v0_8.AMQChannel.deliverCurrentMessageIfComplete(AMQChannel.java:526)
>         at org.apache.qpid.server.protocol.v0_8.AMQChannel.publishContentBody(AMQChannel.java:655)
>         at org.apache.qpid.server.protocol.v0_8.AMQChannel.receiveMessageContent(AMQChannel.java:2519)
>         at org.apache.qpid.framing.ContentBody.process(ContentBody.java:105)
>         at org.apache.qpid.codec.AMQDecoder.processFrame(AMQDecoder.java:394)
>         at org.apache.qpid.server.protocol.v0_8.BrokerDecoder.doProcessFrame(BrokerDecoder.java:114)
>         at org.apache.qpid.server.protocol.v0_8.BrokerDecoder.access$000(BrokerDecoder.java:37)
>         at org.apache.qpid.server.protocol.v0_8.BrokerDecoder$1.run(BrokerDecoder.java:80)
>         at org.apache.qpid.server.protocol.v0_8.BrokerDecoder$1.run(BrokerDecoder.java:76)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.qpid.server.protocol.v0_8.BrokerDecoder.processFrame(BrokerDecoder.java:75)
>         at org.apache.qpid.codec.AMQDecoder.processInput(AMQDecoder.java:370)
>         at org.apache.qpid.codec.AMQDecoder.decodeBuffer(AMQDecoder.java:259)
>         at org.apache.qpid.server.protocol.v0_8.AMQPConnection_0_8$2.run(AMQPConnection_0_8.java:319)
>         at org.apache.qpid.server.protocol.v0_8.AMQPConnection_0_8$2.run(AMQPConnection_0_8.java:299)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:356)
>         at org.apache.qpid.server.protocol.v0_8.AMQPConnection_0_8.received(AMQPConnection_0_8.java:298)
>         at org.apache.qpid.server.transport.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:138)
>         at org.apache.qpid.server.transport.NonBlockingConnection.processAmqpData(NonBlockingConnection.java:465)
>         at org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:45)
>         at org.apache.qpid.server.transport.NonBlockingConnection.processData(NonBlockingConnection.java:398)
>         at org.apache.qpid.server.transport.NonBlockingConnection.readAndProcessData(NonBlockingConnection.java:349)
>         at org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.doRead(NonBlockingConnectionPlainDelegate.java:39)
>         at org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:337)
>         at org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:229)
>         at org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:119)
>         at org.apache.qpid.server.transport.NetworkConnectionScheduler.access$000(NetworkConnectionScheduler.java:37)
>         at org.apache.qpid.server.transport.NetworkConnectionScheduler$2.run(NetworkConnectionScheduler.java:100)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>         at org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.removeMessage(AbstractBDBMessageStore.java:227)
>         ... 37 more
> {noformat}
> Broker log file might have the issue reported in some cases as:
> {noformat}
> 2015-07-23 05:19:30,460 INFO  [default:FileSystemSpaceChecker] (q.m.v.filesystem_full)
- [default:FileSystemSpaceChecker] [vh(/default)/ms(BDBMessageStore)] VHT-1006 : Filesystem
is over 90 per cent full, enforcing flow control.
> 2015-07-23 05:19:46,668 ERROR [IO-/169.124.113.161:58633] (o.a.q.s.u.ServerScopedRuntimeException)
- Unexpected exception occurred on store operation
> com.sleepycat.je.LogWriteException: (JE 5.0.104) Environment must be closed, caused by:
com.sleepycat.je.LogWriteException: Environment invalid because of previous exception: (JE
5.0.104) /apps/qpid/destruction-tests/qpid-broker/6.0.0-SNAPSHOT/work/default/messages java.io.IOException:
No space left on device LOG_WRITE: IOExcep
> {noformat}
> However, in some other cases exception is not get stored in the logs leaving only NPE
(reported into system output) to be the only evidence of an issue which might confuse the
user about the nature of a problem



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Mime
View raw message