Data recovery in distributed storage environments

• US 9,483,367 B1
• Filed: 06/27/2014
• Issued: 11/01/2016
• Est. Priority Date: 06/27/2014
• Status: Active Grant

• Current Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Original Assignee: Veritas Technologies, LLC (Whitehouse Group Ltd.)
• Inventors: Patil, Ram Chandra
• Primary Examiner(s): Manoskey, Joseph D
• Application Number:
  US14/317,645
• Time in Patent Office: 858 Days
• Field of Search: 714/4.21, 714/6.2, 714/4.1, 714/4.11, 714/42, 714/43
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence

Abstract

Various systems and methods for determining whether a distributed storage cluster is operating with a last failure domain are disclosed. The last failure domain is the only node with access to a latest copy of configuration data and an up-to-date copy of application data. In response to a determination that the distributed storage cluster is operating with the last failure domain, the method causes the last failure domain to copy one or more changed blocks of the up-to-date copy of the application data from one or more mirrored volumes to a recovery server for storage.

Claims

1. A method comprising:

determining whether a distributed storage cluster is operating with a last failure domain, wherein

the last failure domain is an only node of a plurality of nodes with access to a latest copy of configuration data and an up-to-date copy of application data; and

in response to a determination that the distributed storage cluster is operating with the last failure domain,

causing the last failure domain to copy one or more changed blocks of the up-to-date copy of the application data from one or more mirrored volumes to a recovery server for storage, wherein

the one or more mirrored volumes are stored at a storage device associated with the last failure domain.

2. The method of claim 1, comprising

causing a master node to copy the latest copy of the configuration data to the recovery server, wherein

the latest copy of the configuration data is for the plurality of nodes.

3. The method of claim 2, comprising:

performing a restore operation, wherein

the restore operation is performed using the latest copy of the configuration data and the one or more changed blocks of the up-to-date copy of application data stored at the recovery server, and

the restore operation is performed on one or more nodes of the plurality of nodes after the one or more nodes rejoin the distributed storage cluster.

4. The method of claim 3, comprising:

deleting the latest copy of the configuration data and the one or more changed blocks of the up-to-date copy of application data from the recovery server after performing the restore operation.

5. The method of claim 3, wherein

the determination of whether the distributed storage cluster is operating under the last failure domain comprises

determining whether all nodes of the plurality of nodes other than the last failure domain have failed,

determining whether the storage device coupled to each node of the plurality nodes has failed, and

determining whether the one or more nodes that rejoin the distributed storage cluster are performing a synchronization operation.

6. The method of claim 3, wherein

the distributed storage cluster maintains a failure count ID that indicates whether the restore operation should be performed from the recovery server or whether the master node comprises the latest copy of the configuration data and the up-to-date copy of the application data.

7. The method of claim 2, wherein

the storage device associated with the master node of the plurality of nodes comprises the up-to-date copy of the application data, and

the master node and the storage device associated with the master node comprise the latest copy of the configuration data, wherein

the master node is the last failure domain,

the master node comprises one part of the latest copy of the configuration data, and

the storage device associated with the master node comprises an another part of the latest copy of the configuration data.

8. The method of claim 1, comprising

determining whether the plurality of nodes are part of the distributed storage cluster.

9. The method of claim 1, wherein

the copying continues until the distributed storage cluster has more than one failure domain.

10. The method of claim 1, wherein

each node of the plurality of nodes is coupled to a corresponding of a plurality of storage devices,

one of more of the corresponding of the plurality of storage devices comprise the one or more mirrored volumes, and

the plurality of nodes execute an application that generates the application data.

11. A non-transitory computer readable storage medium storing program instructions executable to:

determine whether a distributed storage cluster is operating with a last failure domain,

wherein

the last failure domain is an only node of a plurality of nodes with access to a latest copy of configuration data and an up-to-date copy of application data; and

in response to a determination that the distributed storage cluster is operating with the last failure domain,

cause the last failure domain to copy one or more changed blocks of the up-to-date copy of the application data from one or more mirrored volumes to a recovery server for storage, wherein

the one or more mirrored volumes are stored at a storage device associated with the last failure domain.

12. The non-transitory computer readable storage medium of claim 11, comprising:

causing a master node to copy the latest copy of the configuration data to the recovery server, wherein

the latest copy of the configuration data is for the plurality of nodes; and

determining whether the plurality of nodes are part of the distributed storage cluster.

13. The non-transitory computer readable storage medium of claim 12, comprising:

performing a restore operation, wherein

the restore operation is performed using the latest copy of the configuration data and the changed blocks of application data stored at the recovery server, and

the restore operation is performed on one or more nodes of the plurality of nodes after the one or more nodes rejoin the distributed storage cluster.

14. The non-transitory computer readable storage medium of claim 13, wherein

the determination of whether the distributed storage cluster is operating under the last failure domain comprises

determining whether all nodes of the plurality of nodes other than the last failure domain have failed,

determining whether the storage device coupled to each node of the plurality nodes has failed, and

determining whether the one or more nodes that rejoin the distributed storage cluster are performing a synchronization operation.

15. A system comprising:

one or more processors; and

a memory coupled to the one or more processors, wherein the memory stores program, instructions executable by the one or more processors to;

determine whether a distributed storage cluster is operating with a last failure domain,

wherein

the last failure domain is an only node of a plurality of nodes with access to a latest copy of configuration data and an up-to-date copy of application data; and

in response to a determination that the distributed storage cluster is operating with the last failure domain,

cause the last failure domain to copy one or more changed blocks of the up-to-date copy of the application data from one or more mirrored volumes to a recovery server for storage, wherein

the one or more mirrored volumes are stored at a storage device associated with the last failure domain.

16. The system of claim 15, comprising:

performing a restore operation, wherein

the restore operation is performed using the latest copy of the configuration data and the one or more changed blocks of the up-to-date copy of application data stored at the recovery server, and

the restore operation is performed on one or more nodes of the plurality of nodes after the one or more nodes rejoin the distributed storage cluster; and

deleting the latest copy of the configuration data and the one or more changed blocks of the up-to-date copy of application data from the recovery server after performing the restore operation.

17. The system of claim 15, comprising:

causing a master node to copy the latest copy of the configuration data to the recovery server, wherein

the latest copy of the configuration data is for the plurality of nodes.

18. The system of claim 17, wherein

the storage device associated with the master node of the plurality of nodes comprises the up-to-date copy of the application data, and

the master node and the storage device associated with the master node comprise the latest copy of the configuration data, wherein

the master node is the last failure domain,

the master node comprises one part of the latest copy of the configuration data, and

the storage device associated with the master node comprises an another part of the latest copy of the configuration data.

19. The system of claim 17, wherein

the distributed storage cluster maintains a failure count ID that indicates whether the restore operation should be performed from the recovery server or whether the master node comprises the latest copy of the configuration data and the up-to-date copy of the application data.

20. The system of claim 15, wherein

each node of the plurality of nodes is coupled to a corresponding of a plurality of -storage devices,

one of more of the corresponding of the plurality of storage devices comprise the one or more mirrored volumes, and

the plurality of nodes execute an application that generates the application data.