qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lorenz Quack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (QPID-7775) Flow to disk should consider the size of the resident messages in memory.
Date Tue, 16 May 2017 08:48:05 GMT

    [ https://issues.apache.org/jira/browse/QPID-7775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16012003#comment-16012003

Lorenz Quack commented on QPID-7775:

bq. Is it me, or will the way that the AbstractVirtualHost.FlowToDiskCheckingTask cycles through
the queues ensure that queues with a small number of entries will get completely dumped out
to disk before larger queues are fully iterated over?

It was certainly not intended and I don't think it is what is happening.
We iterate over all queues breadth-first (the cycle iterator) and sum up the message sizes.
Only when that cumulative breadth-first sum exceeds the threshold do we start flowing to disk
(i.e., we are flowing the tail of the queues to disk).
So every queue is entitled to the same (+/- 1) number of messages so a long queue would flow
to disk more messages than a short queue.
So the metric of fairness this was designed for is number of in-memory messages on the queue.

The part that I see that is unfair is if the messages on different queues have different sizes.
If queue A has messages with 1 GB while queue B has messages of 1 kB they still get the same
number of in-memory messages meaning queue A will have for example 2 GB (2 messages) while
queue B will only occupy 2 kB (also two messages). 

> Flow to disk should consider the size of the resident messages in memory.
> -------------------------------------------------------------------------
>                 Key: QPID-7775
>                 URL: https://issues.apache.org/jira/browse/QPID-7775
>             Project: Qpid
>          Issue Type: Improvement
>          Components: Java Broker
>            Reporter: Keith Wall
>             Fix For: qpid-java-broker-7.0.0
> Our current algorithm for triggering flow to disk has some shortcomings.
> * the algorithm does not account for memory returned by flow to disk.
> The decision to flow a message a newly arriving message to disk
> considers only the queue's target size and queue's depth.  Once a
> queue depth is over its target, all messages will go to disk even if
> all messages have actually been flowed to disk.  The same is true
> after recovery: flow to disk will be enabled even though there are no
> messages in RAM.
> * the fact that a queue's target size is assigned by periodically by
> housekeeping means that the queues target size can be wrong for most
> of the time.  This is very apparent if a queue is growing; you
> actually see *most* messages flowing to disk even when there is ample
> memory.  The target size is periodically recomputed but only remains
> correct for an instant, the queue returns to the flow to disk state as
> more messages are added.   We see this during perf test runs.
> We will change the MessageStore so that it tracks the size of the messages that are held
resident in memory.    The flow to disk algorithm will be change to by triggered when the
resident memory exceeds the virtual host's target size.
> This work is likely to fully replace the recent work done on QPID-7770.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

View raw message