qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Godfrey <rob.j.godf...@gmail.com>
Subject Re: Proposed QIP: Support for Message Grouping in the broker.
Date Mon, 27 Jun 2011 16:25:18 GMT
So, in general this looks good, and is something we could support in the
Java Broker... My one comment would be that I think the more standard
behaviour for a "Group" is to ensure all messages in the same group are
delivered (in order) to the same consumer, with the notion of blocking
delivery of the nth + 1 element of the group until the nth element is
acknowledged seeming more like an alternate behaviour.

JMS (half) specifies two message properties for group ID and group Sequence
Number... AMQP v1.0 is defining something along the same lines (and as such,
making the attribute containing the group id configurable would make less
sense in an AMQP v1.0 context).  There are some interesting potential
behaviours there for a queue to ensure that messages within a group are not
forwarded "out of order" or with gaps in the sequence...

On the Java side I was planning to implement the more standard Group
behaviour (i.e. messages in a group are all sent to the same consumer), but
could extend that to also do the "don't send until the last message has been
consumed" feature also... we should just work out a clear way of defining
these features and allowing our users to correctly specify which behaviour
it is they desire

Cheers,
Rob

On 24 June 2011 17:07, Ken Giusti <kgiusti@redhat.com> wrote:

>
> Message Groups
>
> Status
>
> Draft
>
> Summary
>
> This document describes a new feature that would allow a message producer
> to enforce the order in which the data from a set of related messages are
> processed by consumers.
>
> Problem
>
> While the broker currently employs an strict FIFO queuing model, it does
> not guarantee that messages will be processed in that order when there are
> multiple consumers subscribed to a queue. Therefore, it is impossible for a
> producer to enforce a strict ordering to the processing of messages, even
> though it may be required by the application.
>
> For example, assume we have a shopping application that manages items in a
> virtual shopping cart. A user may add an item to their shopping cart, then
> change their mind and remove it. If the application sends an "add" message
> to the broker, immediately followed by a "remove" message, they will be
> queued in the proper order - "add", then "remove".
>
> However, if there are multiple consumers, it is possible that once a
> consumer acquires the "add" message, a different consumer may acquire the
> "remove" message. This allows both messages to be processed in parallel,
> which could result in the "remove" operation being performed before the
> "add" operation.
> Solution
>
> This problem can be solved by allowing a producer to mark a group of
> messages as being related, and having the broker enforce strict ordering of
> consumption of messages belonging to that group.
>
> Specifically, for any given group of queued messages, the broker allows
> only the first message in the group be available for consumption by a
> subscriber. The broker blocks the remaining messages belonging to that group
> from consumption by any subscriber. Once the broker has completed the
> transfered of responsibility for that first message to a consumer, it then
> allows the next message in the group to be available for consumption while
> all other messages in that group remain blocked.
>
> In order to guarantee messages are not processed in parallel, the
> application has to ensure that it has completely processed the data in a
> received message before accepting that message, as described in Section
> 2.6.2. Transfer of Responsibility, of the AMQP-0.10 specification.
>
> Note well that distinct message groups would not block each other. For
> example, assume a queue contains messages from two different message groups
> - say group "A" and group "B" - and they are enqueued such that "A"'s
> messages are in front of "B". If the first message of group "A" is in the
> process of being consumed, then the remaining "A" messages are blocked, but
> the first message of the "B" group is available for consumption - even
> though it is "behind" group "A" in the queue.
>
> This feature could be implemented by having the message producer set a
> group identifier in the application headers of a message. Messages belonging
> to the same group would have the same identifier value. The destination
> queue would be configured with the key name of the application header used
> for group identification. When a message is delivered to the queue, the
> broker would check the application headers for the group identification key.
> It would then classify which group the message belonged to based on the
> value of the key. The queue subscriber logic on the broker would need to be
> modified to enforce the rules for accessing grouped messages (as described
> above).
>
> Rationale
>
> The solution described above allows the message producer to enforce a
> strict order to the processing of its messages by the consumers. The broker
> can support such a feature with a minimum of configuration effort (merely
> configuring the header key used per queue).
>
>    * Goal: allow dynamic values for the group identifiers (no
> preconfiguration of identifiers necessary)
>    * Goal: the number of message groups "in flight" should only be limited
> by the available resources (no need to configure a hard limit).
>    * Goal: the individual messages from a single message group may be
> processed by different consumers, as long as the strict order is observed.
>    * Goal: manageability: visibility into the message groups currently on a
> given queue
>    * Goal: manageability: purge, move, reroute messages at the group level.
>    * Goal: "sticky" subscribers - (optional) ensure that all messages from
> a given group are consumed by the same client.
>
> Implementation Notes
>
>    * Queues: support configuration of the group identifier header key.
>    * Messages: provide access to group identifier.
>    * Queues: identify head message of next available message group.
>    * Queues: block the trailing messages in a given message group from
> being consumed.
>    * Consumers: track the message group of the currently acquired
> message(s).
>
> Consequences
>
>    * Development: No changes to the development process.
>    * Release: No changes to the release process.
>    * Documentation: User documentation will be needed to explain the
> feature, and the steps to configure and manage the new feature.
>    * Configuration: Yes: per-queue group identifier header key is
> configurable. Queue state with regard to in-flight message groups needs to
> be visible. Additional methods to purge/move/reroute a message group.
>    * Compatibility: Unlikely - new feature that would need to be enabled
> manually. Applications wishing to use the feature would need to implement a
> message grouping policy, and ensure the processing of received message data
> is completed before signaling acceptance of the message.
>
> References
>
>    * None.
>
> Contributor-in-Charge
>
> Kenneth Giusti, kgiusti@apache.org
> Contributors
>
>    * Ted Ross, tross@redhat.com
>
> Version
>
> 0.1
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project:      http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message