Partition tolerance in cluster membership management

• US 9,672,115 B2
• Filed: 03/13/2014
• Issued: 06/06/2017
• Est. Priority Date: 08/26/2013
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Earl, William, Smith, Gregory A., Knauft, Eric, Karamanolis, Christos, Renauld, Pascal
• Primary Examiner(s): LE, THU NGUYET T
• Application Number:
  US14/209,426
• Publication Number:
  US 20150058298A1
• Time in Patent Office: 1,181 Days
• Field of Search: 707823
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1484 involving virtual machines
• CPC Class Codes: G06F 11/202 where processing functionality is redundant redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: H04L 65/40 Support for services or applications

Abstract

Techniques are disclosed for managing a cluster of computing nodes following a division of the cluster into at least a first and second partition, where the cluster aggregates local storage resources of the nodes to provide an object store, and objects stored in the object store are divided into data components stored across the nodes. In accordance with one method, it is determined that a majority of data components comprising a first object are stored within nodes in the first partition. It is determined that a majority of data components comprising a second object are stored within nodes in the second partition. Configuration objects are permitted to be performed on the first object in the first partition while denying access to the first object from the second partition, and on the second object in the second partition while denying access to the second object from the first partition.

Claims

1. A method comprising:

managing a cluster of computing nodes following a failure or management event that divides the cluster into at least a first partition and a second partition, wherein (i) the cluster aggregates local storage resources of each of the computing nodes to provide an object store, and (ii) each object stored in the object store represents a virtual disk associated with a corresponding virtual machine and contains data that is divided into respective data components stored across the storage resources of the respective computing nodes, wherein managing the cluster of computing nodes comprises;

determining that a majority of data components that comprise a first object stored in the object store are stored within computing nodes in the first partition;

determining that a majority of data components that comprise a second object stored in the object store are stored within computing nodes in the second partition;

in response to determining that the majority of data components that comprise the first object are stored within computing nodes in the first partition, permitting configuration operations to be performed on data components of the first object stored within computing nodes in the first partition while denying any access to the first object from the second partition; and

in response to determining that the majority of data components that comprise the second object are stored within computing nodes in the second partition, permitting configuration operations to be performed on data components of the second object stored within computing nodes in the second partition while denying any access to the second object from the first partition.

2. The method of claim 1, further comprising:

determining that the majority of data components that comprise the first object further includes all data that comprises the first object; and

permitting I/O operations to be performed on the first object in the first partition by a requesting process running on one of the computing nodes in the first partition.

3. The method of claim 2, wherein the requesting process is associated with a first virtual machine and the first object represents a first virtual disk for the first virtual machine.

4. The method of claim 1, wherein configuration operations include operations that modify the capacity, availability, or TOPS of an object stored in the object store.

5. The method of claim 1, wherein each of the computing nodes maintains a local copy of an in-memory database that tracks a current configuration of each object stored in the object store and indicates whether an object is accessible by the computing node.

6. The method of claim 5, further comprising, storing in a log any operations performed on the first object by computing nodes in the first partition.

7. The method of claim 6, wherein, upon a recovery from the failure event, the method further comprises:

replacing the local copies of the in-memory database in the computing nodes of the first partition with a replacement copy of the in-memory database from a leader node of the second partition; and

replaying the operations in the log so that the replacement copy of the in-memory database is updated to include operations performed from the first partition.

8. The method of claim 1, wherein, upon a merging of the first partition and second partition, the method further comprises, reconciling data components of the first object stored within computing nodes in the first partition with any other data component of the first object stored in the second partition.

9. The method of claim 8, wherein the reconciling step further comprises replaying updates made to the first object to a leader node of the merged cluster.

10. The method of claim 1, wherein one or more data components that comprise the first object are stored within computing nodes of the second partition of computing nodes, and wherein denying any access to the first object from the second partition includes denying access to the data components of the first object stored within computing nodes in the second partition.

11. The method of claim 1, comprising:

determining that one or more data components, but less than a majority of data components, that comprise a third object stored the object store are stored within computing nodes in the first partition of computing nodes; and

denying access to the data components of the third object from the first partition.

12. The method of claim 1, wherein the majority of data components that comprise the first object are stored within a plurality of storage resources associated with computing nodes in the first partition.

13. A non-transitory computer readable storage medium storing instructions, which, when executed on a processor, performs operations comprising:

managing a cluster of computing nodes following a failure or management event that divides the cluster into at least a first partition and a second partition, wherein (i) the cluster aggregates local storage resources of each of the computing nodes to provide an object store, and (ii) each object stored in the object store represents a virtual disk associated with a corresponding virtual machine and contains data that is divided into respective data components stored across the storage resources of the respective computing nodes, wherein managing the cluster of computing nodes comprises;

determining that a majority of data components that comprise a first object stored in the object store are stored within computing nodes in the first partition;

determining that a majority of data components that comprise a second object stored in the object store are stored within computing nodes in the second partition;

in response to determining that the majority of data components that comprise the first object are stored within computing nodes in the first partition, permitting configuration operations to be performed on data components of the first object stored within computing nodes in the first partition while denying any access to the first object from the second partition; and

in response to determining that the majority of data components that comprise the second object are stored within computing nodes in the second partition, permitting configuration operations to be performed on data components of the second object stored within computing nodes in the second partition while denying any access to the second object from the first partition.

14. The computer readable storage medium of claim 13, wherein the operation further comprises:

determining that the majority of data components that comprise the first object further includes all data that comprises the first object; and

permitting I/O operations to be performed on the first object in the first partition by a requesting process running on one of the computing nodes in the first partition.

15. The computer readable storage medium of claim 14, wherein the requesting process is associated with a first virtual machine and the first object represents a first virtual disk for the first virtual machine.

16. The computer readable storage medium of claim 13, wherein configuration operations include operations that modify the capacity, availability, or IOPS of an object stored in the object store.

17. The computer readable storage medium of claim 13, wherein each of the computing nodes maintains a local copy of an in-memory database that tracks a current configuration of each object stored in the object store and indicates whether an object is accessible by the computing node.

18. The computer readable storage medium of claim 17, wherein the operation further comprises, storing in a log any operations performed on the first object by computing nodes in the first partition.

19. The computer readable storage medium of claim 18, wherein, upon a recovery from the failure event, the operation further comprises:

replacing the local copies of the in-memory database in the computing nodes of the first partition with a replacement copy of the in-memory database from a leader node of the second partition; and

replaying the operations in the log so that the replacement copy of the in-memory database is updated to include operations performed from the first partition.

20. The computer readable storage medium of claim 13, wherein, upon a merging of the first partition and second partition, the operation further comprises, reconciling data components of the first object stored within computing nodes in the first partition with any other data component of the first object stored in the second partition.

21. The computer readable storage medium of claim 20, wherein the reconciling step further comprises replaying updates made to the first object to a leader node of the merged cluster.

22. A system, comprising:

a processor; and

a memory hosting an application, which, when executed on the processor, performs operations comprising;

managing a cluster of computing nodes following a failure or management event that divides the cluster into at least a first partition and a second partition, wherein (i) the cluster aggregates local storage resources of each of the computing nodes to provide an object store, and (ii) each object stored in the object store represents a virtual disk associated with a corresponding virtual machine and contains data that is divided into respective data components stored across the storage resources of the respective computing nodes, wherein managing the cluster of computing nodes comprises

determining that a majority of data components that comprise a first object stored in the object store are stored within computing nodes in the first partition;

determining that a majority of data components that comprise a second object stored in the object store are stored within computing nodes in the second partition;

in response to determining that the majority of data components that comprise the first object are stored within computing nodes in the first partition, permitting configuration operations to be performed on data components of the first object stored within computing nodes in the first partition while denying any access to the first object from the second partition; and

in response to determining that the majority of data components that comprise the second object are stored within computing nodes in the second partition, permitting configuration operations to be performed on data components of the second object stored within computing nodes in the second partition while denying any access to the second object from the first partition.

23. The system of claim 22, wherein the operation further comprises:

determining that the majority of data components that comprise the first object further includes all data that comprises the first object; and

permitting I/O operations to be performed on the first object in the first partition by a requesting process running on one of the computing nodes in the first partition.

24. The system of claim 23, wherein the requesting process is associated with a first virtual machine and the first object represents a first virtual disk for the first virtual machine.

25. The system of claim 22, wherein configuration operations include operations that modify the capacity, availability, or TOPS of an object stored in the object store.

26. The system of claim 22, wherein each of the computing nodes maintains a local copy of an in-memory database that tracks a current configuration of each object stored in the object store and indicates whether an object is accessible by the computing node.

27. The system of claim 26, wherein the operation further comprises, storing in a log any operations performed on the first object by computing nodes in the first partition.

28. The system of claim 27, wherein upon a recovery from the failure event, the operation further comprises:

replacing the local copies of the in-memory database in the computing nodes of the first partition with a replacement copy of the in-memory database from a leader node of the second partition; and

replaying the operations in the log so that the replacement copy of the in-memory database is updated to include operations performed from the first partition.

29. The system of claim 22, wherein, upon a merging of the first partition and second partition, the operation further comprises, reconciling data components of the first object stored within computing nodes in the first partition with any other data component of the first object stored in the second partition.

30. The system of claim 29, wherein the reconciling step further comprises replaying updates made to the first object to a leader node of the merged cluster.