qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alan Conway <acon...@redhat.com>
Subject Re: [c++ cluster] User doc for persistent clusters.
Date Sat, 12 Dec 2009 03:06:53 GMT
On 12/11/2009 04:58 PM, Sandy Pratt wrote:
>
>
>> -----Original Message-----
>> From: Alan Conway [mailto:aconway@redhat.com]
>> Sent: Friday, December 11, 2009 1:35 PM
>> To: dev@qpid.apache.org
>> Cc: qpid-dev-apache
>> Subject: Re: [c++ cluster] User doc for persistent clusters.
>>
>> On 12/11/2009 03:59 PM, Sandy Pratt wrote:
>>>
>>>> -----Original Message-----
>>>> From: Alan Conway [mailto:aconway@redhat.com]
>>>> Sent: Tuesday, November 24, 2009 7:56 AM
>>>> To: Jonathan Robie; qpid-dev-apache
>>>> Subject: [c++ cluster] User doc for persistent clusters.
>>>>
>>>> I put up a user view of the peristent cluster changes, coming soon.
>>>> Would
>>>> appreaciate any feedback on the doc or the feature it describes.
>>>>
>>>>
>> http://cwiki.apache.org/confluence/display/qpid/Persistent+Cluster+Rest
>>>> art+Design+Note
>>>
>>>
>>> Hi Alan,
>>>
>>> Looks like a great step forward for clustering.  Any hints on what's
>> involved in the manual intervention to enable restart from a full
>> cluster crash?  I'm eager to kick the tires.
>>>
>>
>> Basically it amounts to picking the "best" store and marking it clean
>> by putting
>> a UUID in<data_dir>/cluster/shutdown.uuid, i.e. pretend it was shut
>> down cleanly.
>>
>
> Simple enough, thanks!
>
>> I'm working on providing some help to identify the "best" store and
>> ultimately
>> hope to provide a tool for doing all this a bit more automatically. It
>> will
>> probably mean running the tool on the data-directory of each cluster
>> member
>> initially which is a pain - assumes remote logins or shared file
>> systems.
>>
>> I'd like to find a way to do this from one location without assuming
>> shared
>> filesystems or remote logins. There was a suggestion that if there's a
>> total
>> faliure the brokers come up in "admin mode" where they don't accept any
>> clients
>> except an admin tool. The brokers would collect the info needed to pick
>> the
>> clean store and mark it clean driven remotely by the admin tool. Does
>> this sound
>> like a good direction, or do you have any other suggestions on how to
>> approach this?
>
> That does sound like a good approach to me.
>
> Suppose the cluster has crashed because of a power or network failure (single brokers
crashing due to OS or hardware problems isn't the issue here, if I understand correctly).
>
> Then the common case is that they all power back up without issue, and can unanimously
pick the correct journal while having access to all candidate journals (hand waving a bit
here, maybe*).

In the event of a total failure like this I'm a little uneasy about the cluster 
picking the best store automatically. I had been considering this to be a manual 
intervention case, with some tools that make it fairly trivial but still 
requiring human intervention.

>
> The uncommon case is that they all die for whatever reason, then some member of the cluster
fails to come back up.  At this point, full information from all candidate journals is not
available, and a unanimous decision cannot be reached.  Manual intervention sounds fine here.
>
> *I noticed in some of the JIRA notes that the message store changes are labeled with
a monotonically increasing sequence number.  Is this derived from the Lamport clock implemented
by openais (in which case all the events are conveniently serialized by the CPG)?  I could
be misunderstanding the way the CPG works, but if not it sounds like an excellent way to get
the cluster back in sync.
>
The sequence number is the frame counter maintained by qpidd, but the cluster 
join protocol is designed so that all members will  have the same frame count 
for the same frame regarless when they join the cluster. It's currently not 
persisted so can't be used in recovery for 2 reasons

  - the cost of persisting it every frame time is significant compared to the 
cost to store a message.
  - a frame counter persisted separately from the message store does not 
necessarily reflect the state of the store.

My current thinking is to use a config-change counter (counts membership 
changes) in conjunction with a counter recorded by the store in the same 
transaction with a message called the Persistence ID or PID.
The PID is not the same across the clsuter but but we can resolve that:
  - record the config-change-counter and current PID with every config change.
  - on recovery compare the actual store PID values relative to the PID value 
recorded for the same config change.

I'll be working on this in the coming week or two so any input you have is very 
timely.
Cheers,
Alan.

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org


Mime
View raw message