PARTITION TOLERANCE IN CLUSTER MEMBERSHIP MANAGEMENT

• US 20150058298A1
• Filed: 03/13/2014
• Published: 02/26/2015
• Est. Priority Date: 08/26/2013
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: KARAMANOLIS, Christos, KNAUFT, Eric, RENAULD, Pascal, EARL, William, SMITH, Gregory A.
• Granted Patent: US 9,672,115 B2
• Time in Patent Office: Days
• US Class Current: 707/674
• CPC Class Codes: G06F 11/1425 by reconfiguration of node membership
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1484 involving virtual machines
• CPC Class Codes: G06F 11/202 where processing functionality is redundant redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: H04L 65/40 Support for services or applications

Abstract

Techniques are disclosed for managing a cluster of computing nodes following a division of the cluster into at least a first and second partition, where the cluster aggregates local storage resources of the nodes to provide an object store, and objects stored in the object store are divided into data components stored across the nodes. In accordance with one method, it is determined that a majority of data components comprising a first object are stored within nodes in the first partition. It is determined that a majority of data components comprising a second object are stored within nodes in the second partition. Configuration objects are permitted to be performed on the first object in the first partition while denying access to the first object from the second partition, and on the second object in the second partition while denying access to the second object from the first partition.

Claims

1. A method for managing a cluster of computing nodes following a failure or management event that divides the cluster into at least a first partition and a second partition, wherein (i) the cluster aggregates local storage resources of each of the computing nodes to provide an object store, and (ii) objects stored in the object store are divided into data components stored across the computing nodes, the method comprising:

determining that a majority of data components that comprise a first object stored in the object store are stored within computing nodes in the first partition;

determining that a majority of data components that comprise a second object stored in the object store are stored within computing nodes in the second partition;

permitting configuration operations to be performed on the first object in the first partition while denying any access to the first object from the second partition; and

permitting configuration operations to be performed on the second object in the second partition while denying any access to the second object from the first partition.

2. The method of claim 1, further comprising:

determining that the majority of data components that comprise the first object further includes all data that comprises the first object; and

permitting I/O operations to be performed on the first object in the first partition by a requesting process running on one of the computing nodes in the first partition.

3. The method of claim 2, wherein the requesting process is a virtual machine and the first object represents a virtual disk for the virtual machine.

4. The method of claim 1, wherein configuration operations include operations that modify the capacity, availability, or IOPS of an object stored in the object store.

5. The method of claim 1, wherein each of the computing nodes maintains a local copy of an in-memory database that tracks a current configuration of each object stored in the object store and indicates whether an object is accessible by the computing node.

6. The method of claim 5, further comprising, storing in a log any operations performed on the first object by computing nodes in the first partition.

7. The method of claim 6, wherein, upon a recovery from the failure event, the method further comprises:

replacing the local copies of the in-memory database in the computing nodes of the first partition with a replacement copy of the in-memory database from a leader node of the second partition; and

replaying the operations in the log so that the replacement copy of the in-memory database is updated to include operations performed from the first partition.

8. The method of claim 1, wherein, upon a merging of the first partition and second partition, the method further comprises, reconciliating data components of the first object stored within computing nodes in the first partition with any other data component of the first object stored in the second partition.

9. The method of claim 8, wherein the reconciliating step further comprises replaying updates made to the first object to a leader node of the merged cluster.

10. A non-transitory computer readable storage medium storing instructions, which, when executed on a processor, performs an operation for managing a cluster of computing nodes following a failure or management event that divides the cluster into at least a first partition and a second partition, wherein (i) the cluster aggregates local storage resources of each of the computing nodes to provide an object store, and (ii) objects stored in the object store are divided into data components stored across the computing nodes, the operation comprising:

determining that a majority of data components that comprise a first object stored in the object store are stored within computing nodes in the first partition;

determining that a majority of data components that comprise a second object stored in the object store are stored within computing nodes in the second partition;

permitting configuration operations to be performed on the first object in the first partition while denying any access to the first object from the second partition; and

permitting configuration operations to be performed on the second object in the second partition while denying any access to the second object from the first partition.

11. The computer readable storage medium of claim 10, wherein the operation further comprises:

determining that the majority of data components that comprise the first object further includes all data that comprises the first object; and

permitting I/O operations to be performed on the first object in the first partition by a requesting process running on one of the computing nodes in the first partition.

12. The computer readable storage medium of claim 11, wherein the requesting process is a virtual machine and the first object represents a virtual disk for the virtual machine.

13. The computer readable storage medium of claim 10, wherein configuration operations include operations that modify the capacity, availability, or IOPS of an object stored in the object store.

14. The computer readable storage medium of claim 10, wherein each of the computing nodes maintains a local copy of an in-memory database that tracks a current configuration of each object stored in the object store and indicates whether an object is accessible by the computing node.

15. The computer readable storage medium of claim 14, wherein the operation further comprises, storing in a log any operations performed on the first object by computing nodes in the first partition.

16. The computer readable storage medium of claim 15, wherein, upon a recovery from the failure event, the operation further comprises:

replacing the local copies of the in-memory database in the computing nodes of the first partition with a replacement copy of the in-memory database from a leader node of the second partition; and

replaying the operations in the log so that the replacement copy of the in-memory database is updated to include operations performed from the first partition.

17. The computer readable storage medium of claim 10, wherein, upon a merging of the first partition and second partition, the operation further comprises, reconciliating data components of the first object stored within computing nodes in the first partition with any other data component of the first object stored in the second partition.

18. The computer readable storage medium of claim 17, wherein the reconciliating step further comprises replaying updates made to the first object to a leader node of the merged cluster.

19. A system, comprising:

a processor; and

a memory hosting an application, which, when executed on the processor, performs an operation for managing a cluster of computing nodes following a failure or management event that divides the cluster into at least a first partition and a second partition, wherein (i) the cluster aggregates local storage resources of each of the computing nodes to provide an object store, and (ii) objects stored in the object store are divided into data components stored across the computing nodes, the operation comprising;

determining that a majority of data components that comprise a first object stored in the object store are stored within computing nodes in the first partition;

determining that a majority of data components that comprise a second object stored in the object store are stored within computing nodes in the second partition;

permitting configuration operations to be performed on the first object in the first partition while denying any access to the first object from the second partition; and

permitting configuration operations to be performed on the second object in the second partition while denying any access to the second object from the first partition.

20. The system of claim 19, wherein the operation further comprises:

determining that the majority of data components that comprise the first object further includes all data that comprises the first object; and

permitting I/O operations to be performed on the first object in the first partition by a requesting process running on one of the computing nodes in the first partition.

21. The system of claim 20, wherein the requesting process is a virtual machine and the first object represents a virtual disk for the virtual machine.

22. The system of claim 19, wherein configuration operations include operations that modify the capacity, availability, or IOPS of an object stored in the object store.

23. The system of claim 19, wherein each of the computing nodes maintains a local copy of an in-memory database that tracks a current configuration of each object stored in the object store and indicates whether an object is accessible by the computing node.

24. The system of claim 23, wherein the operation further comprises, storing in a log any operations performed on the first object by computing nodes in the first partition.

25. The system of claim 24, wherein upon a recovery from the failure event, the operation further comprises:

replacing the local copies of the in-memory database in the computing nodes of the first partition with a replacement copy of the in-memory database from a leader node of the second partition; and

replaying the operations in the log so that the replacement copy of the in-memory database is updated to include operations performed from the first partition.

26. The system of claim 19, wherein, upon a merging of the first partition and second partition, the operation further comprises, reconciliating data components of the first object stored within computing nodes in the first partition with any other data component of the first object stored in the second partition.

27. The system of claim 26, wherein the reconciliating step further comprises replaying updates made to the first object to a leader node of the merged cluster.