qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marnie McCormack" <marnie.mccorm...@googlemail.com>
Subject Re: Message history and replay
Date Fri, 19 Jan 2007 15:39:26 GMT
Hi Colin,

Had a little more time to think about this and I believe that there are
potentially two distinct solutions, with pros & cons for each, which
possibly meet the needs of different applications/architectures.

Robert Greig & I discussed briefly and don't entirely agree on the pros/cons
:-)

*Replay with only BDBStore*
If your application has performance as its only really significant
requirement, then a solution which provided a fairly limited set of
operations for BDBStore replaying would be appropriate.

So, you might perhaps have a tool that can archive data (using a header
property) or replay it based on a property, not using SQL since that's not
the BDBStore way.
However, it'd be implementation expensive to try and provide anything more
feature rich.

*Replay with BDBStore and an RDBMS*
So, if you assume that for many users currently replaying data they're
probably using an RDBMS to do it (store/replay/archive/report etc), then
they might find the BDBStore approach fast but not as sophisticated as their
existing architecture.

In that case, it might be nice to be able to setup BDBStore to archive to an
RDBMS and be able to replay from thet RDBMS (mySQL or whatever). The users
can then migrate over any SQL scripts they use to select data from
replay/check the state of their archive/run reports etc - at the cost of a
less performant recovery model.

Could batch up the archive to RDBMS from BDBStore to make it more
efficient/more robust if RDBMS becomes unavailable.

Just some thoughts - hth.

Marnie


On 1/18/07, Colin Crist <colincrist@hermesjms.com> wrote:
>
>
> Typo.
>
> ^On the consumer side, WebMethods (!) for example in their^On the producer
> side, WebMethods (!) for example in their^
>
> Sry. Colin.
>
> > -----Original Message-----
> > From: Colin Crist [mailto:colincrist@hermesjms.com]
> > Sent: 18 January 2007 23:16
> > To: 'qpid-dev@incubator.apache.org'
> > Subject: RE: Message history and replay
> >
> >
> >
> > > > As a user, I would also like to control when messages get
> > > removed as
> > > > part of my end of day or even weekly processs, probably based on
> > > > header property selectors.
> > >
> > > Yes, very interesting and useful use case. It would improve
> > throughput
> > > for many apps since people would not have to commit the
> > whole message
> > > to their database before sending it to the message broker.
> >
> > On the consumer side, WebMethods (!) for example in their
> > core messaging platform can let you find out the last message
> > a client published when it reconnects. I didn't use it when I
> > was in WebMethods hell as our systems did not require such
> > behaviour but I found it interesting and could see scenarios
> > when it may help minimise downsteam duplicates and shorten
> > producer recovery time.
> >
> > I don't think this works for AMQP as it does not maintain the
> > exchange route history - or am I wrong?
> >
> > Personally, I can live without it and its best to keep
> > recovery downstream IMO. It's the consumer end that has to
> > deal with duplicates after all but I've found many systems
> > that on recovery will resend too much - all trades from start
> > of day being a common example that can give unexpected load
> > on the middleware and general network buzz.
> >
> > The lesson I found is that some producers are very simple and
> > when you outsource or buy in products you've just got to live
> > with them!
> >
> > > > I don't believe this is supported in AMQP in the wire
> > protocol (nor
> > > > perhaps should it be) - is it maybe something that could be
> > > done via
> > > > some standardised command messages to a "replay" exchange?
> > >
> > > > Any thoughts on implementation difficulty?
> > >
> > > In terms of our underlying architecture, this would be relatively
> > > easy. We have a delivery table that simply maps from message id to
> > > queue, and the message is stored separately. Messages are removed
> > > automatically when a ref count hits zero. The removal process is
> > > actually handled in the store module so it could do something else
> > > easily (e.g if you had a custom store).
> >
> > > The admin API could be enhanced to support re-enqueuing of
> > message ids
> > > or hard delete of messages.
> >
> > I'd see this kind of behaviour to be message based and
> > perhaps standardised in some addendum to the AMQP
> > specification that defines optional exchanges and payload
> > formats. Of course it makes sense to map it to an API but
> > outside of the core specification to keep it decoupled as
> > much as possible and reduce dependencies and so on. If I were
> > to have a client in a language not in the Qpid toolkit, it
> > would be good to have the messages and interaction well
> > defined so another implementation could leverage and help
> > drive it into a standard.
> >
> > Its hardly new and there are always things to learn from the
> > old dogs of middleware (with most of the current market share...)
> >
> > http://www-1.ibm.com/support/docview.wss?rs=203&uid=swg2400066
> > 8&loc=en_US&cs=utf-8&lang=en
> >
> > The same argument goes for configuration, management and
> > monitoring functionality too :) I'd like to see a market for
> > 3rd party tooling that works with all AMQP based products. A
> > whole other conversation when you and the spec are hitting 1.0...
> >
> > Hope this is not too far leftfield for this stage of the game
> > and you don't mind me throwing in my opinions...
> >
> > Regards,
> >
> > Colin.
> >
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message