Fast primary cluster recovery

• US 7,917,469 B2
• Filed: 11/07/2007
• Issued: 03/29/2011
• Est. Priority Date: 11/08/2006
• Status: Active Grant

• Current Assignee: Hitachi Vantara, LLC (Hitachi, Ltd.)
• Original Assignee: Hitachi Data Systems Corporation (Hitachi, Ltd.)
• Inventors: Bernhard, Benjamin K. D., Mason, Robert S. Jr.
• Primary Examiner(s): AL HASHEMI, SANA A
• Application Number:
  US11/936,317
• Publication Number:
  US 20090006888A1
• Time in Patent Office: 1,238 Days
• Field of Search: 707/610
• US Class Current: 707/610
• CPC Class Codes: G06F 11/08 Error detection or correction by redundancy in data representation, e.g. by using checking codes
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups

Abstract

A cluster recovery process is implemented across a set of distributed archives, where each individual archive is a storage cluster of preferably symmetric nodes. Each node of a cluster typically executes an instance of an application that provides object-based storage of fixed content data and associated metadata. According to the storage method, an association or “link” between a first cluster and a second cluster is first established to facilitate replication. The first cluster is sometimes referred to as a “primary” whereas the “second” cluster is sometimes referred to as a “replica.” Once the link is made, the first cluster'"'"'s fixed content data and metadata are then replicated from the first cluster to the second cluster, preferably in a continuous manner. Upon a failure of the first cluster, however, a failover operation occurs, and clients of the first cluster are redirected to the second cluster. Upon repair or replacement of the first cluster (a “restore”), the repaired or replaced first cluster resumes authority for servicing the clients of the first cluster. This restore operation preferably occurs in two stages: a “fast recovery” stage that involves preferably “bulk” transfer of the first cluster metadata, following by a “fail back” stage that involves the transfer of the fixed content data. Upon receipt of the metadata from the second cluster, the repaired or replaced first cluster resumes authority for the clients irrespective of whether the fail back stage has completed or even begun.

Claims

1. A storage method operating across a set of distributed locations, wherein at each location a redundant array of independent nodes are networked together to provide a cluster, and wherein each node of a cluster executes an instance of an application that provides object-based storage of fixed content data and associated metadata, comprising:

configuring an association between a first cluster and a second cluster;

replicating the first cluster'"'"'s fixed content data and metadata from the first cluster to the second cluster;

upon a failure associated with the first cluster, redirecting clients of the first cluster to the second cluster; and

upon repair or replacement of the first cluster, having the repaired or replaced first cluster resume authority for servicing the clients of the first cluster upon receipt, from the second cluster, of the metadata, wherein the repaired or replaced first cluster resumes authority for the clients irrespective of whether the fixed content data has been transferred back from the second cluster.

2. The storage method as described in claim 1 wherein, during the replicating step, the first cluster'"'"'s metadata is transferred from the first cluster to second cluster prior to the fixed content data.

3. The storage method as described in claim 1 wherein, upon repair or replacement of the first cluster, the first cluster'"'"'s metadata is transferred from the second cluster to the repaired or replaced first cluster prior to the fixed content data.

4. The storage method as described in claim 3 wherein the metadata is transferred from the second cluster to the repaired or replaced first cluster by a bulk transfer process.

5. The storage method as described in claim 1 further including directing clients of the first cluster to obtain fixed content data from nodes in the repaired or replaced first cluster once the fixed content data has been restored to the repaired or replaced first cluster.

6. In a distributed storage system wherein a first cluster replicates metadata and content data to a second cluster, a method, comprising:

upon a failure associated with the first cluster, redirecting clients of the first cluster to the second cluster; and

bulk transferring the metadata to a repaired or replaced first cluster in advance of any transfer of the content data, the repaired or replaced first cluster resuming authority for servicing the clients of the first cluster as soon as the bulk transfer of the metadata is completed.

7. The method as described in claim 6 further including restoring the content data to the repaired or replaced first cluster following bulk transfer of the metadata.

8. The method as described in claim 7 wherein, in response to a client request, a read is performed in the first cluster and an associated content retrieval is performed in the second cluster until the content data has been restored to the repaired or replaced first cluster.

9. The method as described in claim 6 wherein the metadata and content data are associated with a namespace for which the first cluster is authoritative.

10. The method as described in claim 6 wherein each of the first and second clusters is a redundant array of independent nodes networked together, wherein each node executes an instance of an application that provides object-based storage of the content data and the metadata.

11. The method as described in claim 10 wherein the content data is fixed content data.