Method and system for managing metadata for a virtualization environment

• US 9,747,287 B1
• Filed: 08/01/2012
• Issued: 08/29/2017
• Est. Priority Date: 08/10/2011
• Status: Active Grant

• Current Assignee: Nutanix Incorporated
• Original Assignee: Nutanix Incorporated
• Inventors: Bhardwaj, Rishi, Guturi, Venkata Ranga Radhanikanth, Aron, Mohit
• Primary Examiner(s): Yi, David X
• Assistant Examiner(s): Goldschmidt, Craig
• Application Number:
  US13/564,511
• Time in Patent Office: 1,854 Days
• Field of Search: None
• CPC Class Codes: G06F 16/2308 Concurrency control transaction processing G06F9/466
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays

Abstract

Disclosed is an improved approach for managing updates to metadata for a virtualization environment. According to some embodiments, a compare and swap approach is taken to manage updates and to handle possible inconsistencies.

Claims

1. A computer implemented method for managing metadata in a virtualization system, comprising:

identifying metadata to track data within storage devices for the virtualization system, the virtualization system comprising a plurality of nodes that includes at least a first node having a first local storage device and a first service virtual machine and a second node having a second local storage device and a second service virtual machine, the first local storage device and the second local storage device forming part of a global storage pool such that any of the first node and the second node can access virtual disks on both the first local storage device and the second local storage device, with the first node and the second node concurrently accessing the metadata through their respective service virtual machines to manage the virtual disks within the global storage pool the virtual disks being virtual structures implemented by the metadata managed by the first and second service virtual machines;

performing a compare and swap operation to manage an update to the metadata maintained by service virtual machines on each of the plurality of nodes in the virtualization system, the compare and swap operation addressing possible data inconsistencies caused by multiple update operations generated by the plurality of nodes when concurrently updating the metadata for the data in the global storage pool having the first local storage device and the second local storage device that are accessible by both the first node and the second node through their respective service virtual machines, wherein validation information for the metadata is processed by the first or second service virtual machines to perform the update;

receiving, from a node from among the plurality of nodes, an instruction to perform a read operation;

before performing the read operation, determining whether the plurality of nodes comprises a downed node;

if the plurality of nodes comprises the downed node, then incrementing a current consensus instance to a new instance count and choosing a value for a prior instance to be associated with the new instance count; and

invalidating a write operation for the downed node if the downed node comes back up, wherein the write operation for the downed node is invalidated as a result of a consensus instance corresponding to the write operation being different than the new instance count.

2. The method of claim 1, wherein the validation information comprises a timestamp or a checksum.

3. The method of claim 1, wherein the validation information is obtained for the metadata and is checked prior to completion of the update.

4. The method of claim 1, wherein the compare and swap operation uses a consensus algorithm to manage the update to the metadata maintained on the plurality of nodes.

5. The method of claim 4, in which the consensus algorithm provides for selection of a value by consensus among the plurality of nodes, wherein a consensus instance is incremented to a new value.

6. The method of claim 1, in which an inconsistency is identified when a previously downed node that previously made the update has an instance value different from the value associated with the new instance count.

7. The method of claim 6, in which the update by the previously downed node is invalidated.

8. The method of claim 4, in which a sequence of updates is identified with an epoch value.

9. The method of claim 8, in which the epoch value is used to process a deletion of data.

10. The method of claim 1, in which a race condition is identified and accessing of the metadata is restarted, wherein the race condition is identified because of an inconsistency in the metadata.

11. The method of claim 10, in which a check is made whether the inconsistency is resolved after accessing of the metadata is restarted, wherein an error condition is identified if the inconsistency is not resolved.

12. The method of claim 1 in which an update operation to an item of metadata is performed such that the item of metadata is confirmed not to have been changed by another entity from a start of the update operation to a conclusion of the update operation.

13. The method of claim 1, in which a tabular structure is used to maintain entries for metadata values, and an instance count corresponds to the metadata values in the tabular structure, wherein the instance count is increased for updates to the metadata values.

14. The method of claim 13, in which a consolidated instance count is maintained for multiple entries in the tabular structure, the consolidated instance count corresponding to a collective set of data.

15. The method of claim 14, in which the consolidated instance count comprises a row instance value for a row of entries.

16. The method of claim 15, further comprising:

determining whether the plurality of nodes comprises a downed node and incrementing the consolidated instance count to a new instance count and setting a delete clock if the plurality of nodes comprises the downed node, wherein the delete clock is maintained for the tabular structure; and

invalidating any changes made by the downed node to the tabular structure if the downed node comes back up, wherein a column from the tabular structure associated with a time less than the delete clock is invalidated or deleted.

17. The method of claim 16, wherein incrementing the consolidated instance count further comprises copying contents of an entire row for the new instance.

18. A computer program product embodied on a non-transitory computer readable medium, the non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a method for managing metadata in a virtualization system, the method comprising:

identifying metadata to track data within storage devices for the virtualization system, the virtualization system comprising a plurality of nodes that includes at least a first node having a first local storage device and a first service virtual machine and a second node having a second local storage device and a second service virtual machine, the first local storage device and the second local storage device forming part of a global storage pool such that any of the first node and the second node can access virtual disks on both the first local storage device and the second local storage device, with the first node and the second node concurrently accessing the metadata through their respective service virtual machines to manage the virtual disks within the global storage pool the virtual disks being virtual structures implemented by the metadata managed by the first and second service virtual machines;

performing a compare and swap operation to manage an update to the metadata maintained by service virtual machines on each of the plurality of nodes in the virtualization system, the compare and swap operation addressing possible data inconsistencies caused by multiple update operations generated by the plurality of nodes when concurrently updating the metadata for the data in the global storage pool having the first local storage device and the second local storage device that are accessible by both the first node and the second node through their respective service virtual machines, wherein validation information for the metadata is processed by the first or second service virtual machines to perform the update;

receiving, from a node from among the plurality of nodes, an instruction to perform a read operation;

before performing the read operation, determining whether the plurality of nodes comprises a downed node;

if the plurality of nodes comprises the downed node, then incrementing a current consensus instance to a new instance count and choosing a value for a prior instance to be associated with the new instance count; and

invalidating a write operation for the downed node if the downed node comes back up, wherein the write operation for the downed node is invalidated as a result of a consensus instance corresponding to the write operation being different than the new instance count.

19. The computer program product of claim 18, wherein the validation information comprises a timestamp or a checksum.

20. The computer program product of claim 18, wherein the validation information is obtained for the metadata and is checked prior to completion of the update.

21. The computer program product of claim 18, wherein the compare and swap operation uses a consensus algorithm to manage the update to the metadata maintained on the plurality of nodes.

22. The computer program product of claim 21, in which the consensus algorithm provides for selection of a value by consensus among the plurality of nodes, wherein a consensus instance is incremented to a new value.

23. The computer program product of claim 18, in which an inconsistency is identified when a previously downed node that previously made the update has an instance value different from the value associated with the new instance count.

24. The computer program product of claim 23, in which the update by the previously downed node is invalidated.

25. The computer program product of claim 21, in which a sequence of updates is identified with an epoch value.

26. The computer program product of claim 25, in which the epoch value is used to process a deletion of data.

27. The computer program product of claim 18, in which a race condition is identified and accessing of the metadata is restarted, wherein the race condition is identified because of an inconsistency in the metadata.

28. The computer program product of claim 27, in which a check is made whether the inconsistency is resolved after accessing of the metadata is restarted, wherein an error condition is identified if the inconsistency is not resolved.

29. The computer program product of claim 18, in which an update operation to an item of metadata is performed such that the item of metadata is confirmed not to have been changed by another entity from a start of the update operation to a conclusion of the update operation.

30. The computer program product of claim 18, in which a tabular structure is used to maintain entries for metadata values, and an instance count corresponds to the metadata values in the tabular structure, wherein the instance count is increased for updates to the metadata values.

31. The computer program product of claim 30, in which a consolidated instance count is maintained for multiple entries in the tabular structure.

32. The computer program product of claim 31, in which the consolidated instance count comprises a row instance value for a row of entries.

33. The computer program product of claim 32, further comprising:

determining whether the plurality of nodes comprises a downed node and incrementing the consolidated instance count to a new instance count and setting a delete clock if the plurality of nodes comprises the downed node, wherein the delete clock is maintained for the tabular structure; and

invalidating any changes made by the downed node to the tabular structure if the downed node comes back up, wherein a column from the tabular structure associated with a time less than the delete clock is invalidated or deleted.

34. The computer program product of claim 33, wherein a column associated with a time less than the delete clock is invalidated or deleted.

35. A storage system comprising storage devices in a virtualization system, comprising:

a processor to handle computing instructions to access the storage devices; and

computer readable medium comprising identifying metadata to track data within storage devices for the virtualization system, the virtualization system comprising a plurality of nodes that includes at least a first node having a first local storage device and a first service virtual machine and a second node having a second local storage device and a second service virtual machine, the first local storage device and the second local storage device forming part of a global storage pool such that any of the first node and the second node can access virtual disks on both of the first local storage device and the second local storage device, with the first node and the second node concurrently accessing the metadata through their respective service virtual machines to manage the virtual disks within the global storage pool, the virtual disks being virtual structures implemented by the metadata managed by the first and second service virtual machines,

where a compare and swap operation is performed to manage an update to the metadata maintained by service virtual machines on each of the plurality of nodes in the virtualization system, the compare and swap operation addressing possible data inconsistencies caused by multiple update operations generated by the plurality of nodes when concurrently updating the metadata for the data in the global storage pool having the first local storage device and the second local storage device that are accessible by both the first node and the second node through their respective service virtual machines, wherein validation information for the metadata is processed by the first or second service virtual machines to perform the update;

receiving, from a node from among the plurality of nodes, an instruction to perform a read operation;

before performing the read operation determining whether the plurality of nodes comprises a downed node;

if the plurality of nodes comprises the downed node, then incrementing a current consensus instance to a new instance count and choosing a value for a prior instance to be associated with the new instance count; and

invalidating a write operation for the downed node if the downed node comes back up, wherein the write operation for the downed node is invalidated as a result of a consensus instance corresponding to the write operation being different than the new instance count.

36. The system of claim 35, wherein the validation information comprises a timestamp or a checksum.

37. The system of claim 35, wherein the validation information is obtained for the metadata and is checked prior to completion of the update.

38. The system of claim 35, wherein the compare and swap operation uses a consensus algorithm to manage the update to the metadata maintained on the plurality of nodes.

39. The system of claim 38, in which the consensus algorithm provides for selection of a value by consensus among the plurality of nodes, wherein a consensus instance is incremented to a new value.

40. The system of claim 35, in which an inconsistency is identified when a previously downed node that previously made the update has an instance value different from the value associated with the new instance count.

41. The system of claim 40, in which the update by the previously downed node is invalidated.

42. The system of claim 38, in which a sequence of updates corresponds to an epoch value.

43. The system of claim 42, in which the epoch value corresponds to a deletion of data.

44. The system of claim 35, in which identification of a race condition corresponds to restart of the metadata, wherein the race condition is identified because of an inconsistency in the metadata.

45. The system of claim 35, further comprising a tabular structure to maintain entries for metadata values, wherein an instance count corresponds to the metadata values in the tabular structure, and wherein the instance count is increased for updates to the metadata values.

46. The system of claim 45, in which a consolidated instance count is maintained for multiple entries in the tabular structure, the consolidated instance count corresponding to a collective set of data.

47. The system of claim 46, in which the consolidated instance count comprises a row instance value for a row of entries.

48. The system of claim 47, further comprising:

determining whether the plurality of nodes comprises a downed node and incrementing the consolidated instance count to a new instance count and setting a delete clock if the plurality of nodes comprises the downed node, wherein the delete clock is maintained for the tabular structure; and

invalidating any changes made by the downed node to the tabular structure if the downed node comes back up, wherein a column from the tabular structure associated with a time less than the delete clock is invalidated or deleted.

49. The system of claim 48, wherein incrementing the consolidated instance count further comprises copying contents of an entire row for the new instance.