qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Crist" <colincr...@hermesjms.com>
Subject RE: Message history and replay
Date Thu, 18 Jan 2007 23:16:03 GMT
 

> > As a user, I would also like to control when messages get 
> removed as 
> > part of my end of day or even weekly processs, probably based on 
> > header property selectors.
> 
> Yes, very interesting and useful use case. It would improve 
> throughput for many apps since people would not have to 
> commit the whole message to their database before sending it 
> to the message broker.

On the consumer side, WebMethods (!) for example in their core messaging
platform can let you find out the last message a client published when it
reconnects. I didn't use it when I was in WebMethods hell as our systems did
not require such behaviour but I found it interesting and could see
scenarios when it may help minimise downsteam duplicates and shorten
producer recovery time.

I don't think this works for AMQP as it does not maintain the exchange route
history - or am I wrong?

Personally, I can live without it and its best to keep recovery downstream
IMO. It's the consumer end that has to deal with duplicates after all but
I've found many systems that on recovery will resend too much - all trades
from start of day being a common example that can give unexpected load on
the middleware and general network buzz. 

The lesson I found is that some producers are very simple and when you
outsource or buy in products you've just got to live with them!

> > I don't believe this is supported in AMQP in the wire protocol (nor 
> > perhaps should it be) - is it maybe something that could be 
> done via 
> > some standardised command messages to a "replay" exchange?
> 
> > Any thoughts on implementation difficulty?
> 
> In terms of our underlying architecture, this would be 
> relatively easy. We have a delivery table that simply maps 
> from message id to queue, and the message is stored 
> separately. Messages are removed automatically when a ref 
> count hits zero. The removal process is actually handled in 
> the store module so it could do something else easily (e.g if 
> you had a custom store).

> The admin API could be enhanced to support re-enqueuing of 
> message ids or hard delete of messages.

I'd see this kind of behaviour to be message based and perhaps standardised
in some addendum to the AMQP specification that defines optional exchanges
and payload formats. Of course it makes sense to map it to an API but
outside of the core specification to keep it decoupled as much as possible
and reduce dependencies and so on. If I were to have a client in a language
not in the Qpid toolkit, it would be good to have the messages and
interaction well defined so another implementation could leverage and help
drive it into a standard.

Its hardly new and there are always things to learn from the old dogs of
middleware (with most of the current market share...)

http://www-1.ibm.com/support/docview.wss?rs=203&uid=swg24000668&loc=en_US&cs
=utf-8&lang=en

The same argument goes for configuration, management and monitoring
functionality too :) I'd like to see a market for 3rd party tooling that
works with all AMQP based products. A whole other conversation when you and
the spec are hitting 1.0...

Hope this is not too far leftfield for this stage of the game and you don't
mind me throwing in my opinions...

Regards,

Colin.



Mime
View raw message