qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marnie McCormack (JIRA)" <qpid-...@incubator.apache.org>
Subject [jira] Updated: (QPID-32) Amend persistence algorithm to introduce paging to physical memory
Date Tue, 31 Oct 2006 14:58:17 GMT
     [ http://issues.apache.org/jira/browse/QPID-32?page=all ]

Marnie McCormack updated QPID-32:

    Fix Version/s: M2

> Amend persistence algorithm to introduce paging to physical memory
> ------------------------------------------------------------------
>                 Key: QPID-32
>                 URL: http://issues.apache.org/jira/browse/QPID-32
>             Project: Qpid
>          Issue Type: Improvement
>          Components: Java Broker
>            Reporter: Marnie McCormack
>         Assigned To: Robert Greig
>             Fix For: M2
> Currently all messages, including the headers and body, are held in memory in the broker.
This means that even with persistent queues, all messages must be able to fit in the heap.
Clearly that is a severe limitation. 
> This effectively means that there is an upper limit for the number of messages a broker
can handle, bound by the heap size available and defined by the size of the messages in transit.
> As we know that there are users who wish to combine high volume throughput with large
message size (100Mb) we can expect that this limitation will be constraining fairly quickly,
given finite available memory. We have had requests for the broker to be able to handle 5
million messages at any one time, with message size being frmo small (a few kb) to large (hundreds
of Mb).
> We need to introduce a paging algorithm into the broker such that it can page messages
in & out of memory as appropriate during transit. The broker needs to be enhanced so that
messages do not need to be held in memory at all times. Ideally the redesign of this aspect
will support flow-to-disk for non-persistent messages when configurable thresholds are reached.
> This may also affect transactional messages, persistent or not, as they are also held
in memory during transit and thus are affected by the same constraint.
> The design of the solution for memory management is critical for the performance of the
broker and should be published and reviewed, as well as having defined performance benchmarks
to meet during testing. We need to be very confident that we're putting in a solution which
is quick and efficient.
> We may also want to consider the configuration options available for memory management
- allowing our users to tune deployed brokers in a way appropriate for their application profile
i.e. if the app is sending large messages infrequently then the fastest solution could potentially
differ from the appropriate model for an application sending mainly small messages and occasional
very large messages.
> Also need to consider the impact of priority/qos settings on messages when designing
the solution. 
> The assignee for this work should expand the requirements in more detail once analysis
is complete, prior to completion of development.
> Please add/amend the detail here if you have hands on experience of the existing persistence
code. Thanks.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message