qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Conway <acon...@redhat.com>
Subject Re: Proposed QIP: Support for Message Grouping in the broker.
Date Mon, 27 Jun 2011 10:32:13 GMT
On 06/24/2011 11:07 AM, Ken Giusti wrote:
> Message Groups
> Status
> Draft
> Summary
> This document describes a new feature that would allow a message producer to enforce
the order in which the data from a set of related messages are processed by consumers.
> Problem
> While the broker currently employs an strict FIFO queuing model, it does not guarantee
that messages will be processed in that order when there are multiple consumers subscribed
to a queue. Therefore, it is impossible for a producer to enforce a strict ordering to the
processing of messages, even though it may be required by the application.
> For example, assume we have a shopping application that manages items in a virtual shopping
cart. A user may add an item to their shopping cart, then change their mind and remove it.
If the application sends an "add" message to the broker, immediately followed by a "remove"
message, they will be queued in the proper order - "add", then "remove".
> However, if there are multiple consumers, it is possible that once a consumer acquires
the "add" message, a different consumer may acquire the "remove" message. This allows both
messages to be processed in parallel, which could result in the "remove" operation being performed
before the "add" operation.
> Solution
> This problem can be solved by allowing a producer to mark a group of messages as being
related, and having the broker enforce strict ordering of consumption of messages belonging
to that group.
> Specifically, for any given group of queued messages, the broker allows only the first
message in the group be available for consumption by a subscriber. The broker blocks the remaining
messages belonging to that group from consumption by any subscriber. Once the broker has completed
the transfered of responsibility for that first message to a consumer, it then allows the
next message in the group to be available for consumption while all other messages in that
group remain blocked.
> In order to guarantee messages are not processed in parallel, the application has to
ensure that it has completely processed the data in a received message before accepting that
message, as described in Section 2.6.2. Transfer of Responsibility, of the AMQP-0.10 specification.
> Note well that distinct message groups would not block each other. For example, assume
a queue contains messages from two different message groups - say group "A" and group "B"
- and they are enqueued such that "A"'s messages are in front of "B". If the first message
of group "A" is in the process of being consumed, then the remaining "A" messages are blocked,
but the first message of the "B" group is available for consumption - even though it is "behind"
group "A" in the queue.
> This feature could be implemented by having the message producer set a group identifier
in the application headers of a message. Messages belonging to the same group would have the
same identifier value. The destination queue would be configured with the key name of the
application header used for group identification. When a message is delivered to the queue,
the broker would check the application headers for the group identification key. It would
then classify which group the message belonged to based on the value of the key. The queue
subscriber logic on the broker would need to be modified to enforce the rules for accessing
grouped messages (as described above).
> Rationale
> The solution described above allows the message producer to enforce a strict order to
the processing of its messages by the consumers. The broker can support such a feature with
a minimum of configuration effort (merely configuring the header key used per queue).
>      * Goal: allow dynamic values for the group identifiers (no preconfiguration of identifiers
>      * Goal: the number of message groups "in flight" should only be limited by the available
resources (no need to configure a hard limit).
>      * Goal: the individual messages from a single message group may be processed by
different consumers, as long as the strict order is observed.
>      * Goal: manageability: visibility into the message groups currently on a given queue
>      * Goal: manageability: purge, move, reroute messages at the group level.
>      * Goal: "sticky" subscribers - (optional) ensure that all messages from a given
group are consumed by the same client.
> Implementation Notes
>      * Queues: support configuration of the group identifier header key.
>      * Messages: provide access to group identifier.
>      * Queues: identify head message of next available message group.
>      * Queues: block the trailing messages in a given message group from being consumed.
>      * Consumers: track the message group of the currently acquired message(s).
> Consequences
>      * Development: No changes to the development process.
>      * Release: No changes to the release process.
>      * Documentation: User documentation will be needed to explain the feature, and the
steps to configure and manage the new feature.
>      * Configuration: Yes: per-queue group identifier header key is configurable. Queue
state with regard to in-flight message groups needs to be visible. Additional methods to purge/move/reroute
a message group.
>      * Compatibility: Unlikely - new feature that would need to be enabled manually.
Applications wishing to use the feature would need to implement a message grouping policy,
and ensure the processing of received message data is completed before signaling acceptance
of the message.

I think we need to look at refactoring the queue so that functionality like this 
can be added as a set of plug-in modules rather than the existing 
swiss-army-knife approach. I don't think the existing approach will scale from a 
maintenance point of view as we add more optional features to the queue.

Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org

View raw message