qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marnie McCormack (JIRA)" <qpid-...@incubator.apache.org>
Subject [jira] Created: (QPID-32) Amend persistence algorithm to introduce paging to physical memory
Date Thu, 12 Oct 2006 12:33:34 GMT
Amend persistence algorithm to introduce paging to physical memory 

                 Key: QPID-32
                 URL: http://issues.apache.org/jira/browse/QPID-32
             Project: Qpid
          Issue Type: Improvement
          Components: Java Broker
            Reporter: Marnie McCormack

At present all persistent messages are held in memory when being acted upon by the broker.

This effectively means that there is an upper limit for the number of messages a broker can
handle, bound by the heap size available and defined by the size of the messages in transit.

As we know that there are users who wish to combine high volume throughput with large message
size (100Mb) we can expect that this limitation will be constraining fairly quickly, given
finite available memory.

We need to introduce a paging algorithm into the broker such that it can page messages in
& out of memory as appropriate during transit. 

This may also affect transactional messages, persistent or not, as they are also held in memory
during transit and thus are affected by the same constraint.

The design of the solution for memory management is critical for the performance of the broker
and should be published and reviewed, as well as having defined performance benchmarks to
meet during testing. We need to be very confident that we're putting in a solution which is
quick and efficient.

We may also want to consider the configuration available for memory management - allowing
our users to tune deployed brokers in a way appropriate for their application profile i.e.
if the app is sending large messages infrequently then the fastest solution could potentially
differ from the appropriate model for an application sending mainly small messages and occasional
very large messages.

Also need to consider the impact of priority/qos settings on messages when designing the solution.

The assignee for this work should expand the requirements in more detail once analysis is
complete, prior to completion of development.

Please add/amend the detail here if you have hands on experience of the existing persistence
code. Thanks.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message