qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy Pratt <prat...@adobe.com>
Subject RE: [c++ cluster] User doc for persistent clusters.
Date Fri, 11 Dec 2009 21:58:06 GMT


> -----Original Message-----
> From: Alan Conway [mailto:aconway@redhat.com]
> Sent: Friday, December 11, 2009 1:35 PM
> To: dev@qpid.apache.org
> Cc: qpid-dev-apache
> Subject: Re: [c++ cluster] User doc for persistent clusters.
> 
> On 12/11/2009 03:59 PM, Sandy Pratt wrote:
> >
> >> -----Original Message-----
> >> From: Alan Conway [mailto:aconway@redhat.com]
> >> Sent: Tuesday, November 24, 2009 7:56 AM
> >> To: Jonathan Robie; qpid-dev-apache
> >> Subject: [c++ cluster] User doc for persistent clusters.
> >>
> >> I put up a user view of the peristent cluster changes, coming soon.
> >> Would
> >> appreaciate any feedback on the doc or the feature it describes.
> >>
> >>
> http://cwiki.apache.org/confluence/display/qpid/Persistent+Cluster+Rest
> >> art+Design+Note
> >
> >
> > Hi Alan,
> >
> > Looks like a great step forward for clustering.  Any hints on what's
> involved in the manual intervention to enable restart from a full
> cluster crash?  I'm eager to kick the tires.
> >
> 
> Basically it amounts to picking the "best" store and marking it clean
> by putting
> a UUID in <data_dir>/cluster/shutdown.uuid, i.e. pretend it was shut
> down cleanly.
> 

Simple enough, thanks!

> I'm working on providing some help to identify the "best" store and
> ultimately
> hope to provide a tool for doing all this a bit more automatically. It
> will
> probably mean running the tool on the data-directory of each cluster
> member
> initially which is a pain - assumes remote logins or shared file
> systems.
> 
> I'd like to find a way to do this from one location without assuming
> shared
> filesystems or remote logins. There was a suggestion that if there's a
> total
> faliure the brokers come up in "admin mode" where they don't accept any
> clients
> except an admin tool. The brokers would collect the info needed to pick
> the
> clean store and mark it clean driven remotely by the admin tool. Does
> this sound
> like a good direction, or do you have any other suggestions on how to
> approach this?

That does sound like a good approach to me.

Suppose the cluster has crashed because of a power or network failure (single brokers crashing
due to OS or hardware problems isn't the issue here, if I understand correctly).

Then the common case is that they all power back up without issue, and can unanimously pick
the correct journal while having access to all candidate journals (hand waving a bit here,
maybe*).

The uncommon case is that they all die for whatever reason, then some member of the cluster
fails to come back up.  At this point, full information from all candidate journals is not
available, and a unanimous decision cannot be reached.  Manual intervention sounds fine here.

*I noticed in some of the JIRA notes that the message store changes are labeled with a monotonically
increasing sequence number.  Is this derived from the Lamport clock implemented by openais
(in which case all the events are conveniently serialized by the CPG)?  I could be misunderstanding
the way the CPG works, but if not it sounds like an excellent way to get the cluster back
in sync.

Sandy


---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org


Mime
View raw message