Method and system for maintaining consistency for I/O operations on metadata distributed amongst nodes in a ring structure

• US 10,379,759 B2
• Filed: 08/16/2017
• Issued: 08/13/2019
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Nutanix Incorporated
• Original Assignee: Nutanix Incorporated
• Inventors: Bhardwaj, Rishi, Guturi, Venkata Ranga Radhanikanth
• Primary Examiner(s): Woo, Isaac M
• Application Number:
  US15/678,893
• Publication Number:
  US 20170364290A1
• Time in Patent Office: 727 Days
• Field of Search: 707600-899
• CPC Class Codes: G06F 16/188 Virtual file systems
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0614 Improving the reliability of storage systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A method for maintaining consistency for I/O operations on metadata distributed amongst a plurality of nodes forming a ring structure.

Claims

1. A method, comprising:

maintaining a network virtualization environment comprising a plurality of nodes forming a ring structure, wherein a storage pool is formed from locally attached storage devices from the plurality of nodes;

adding a new node that is adjacent to an existing node in the plurality of nodes forming the ring structure, wherein the new node is placed in a state that identifies the new node for non-inclusion in a quorum;

receiving an I/O request for data in the storage pool stored at the existing node;

and

establishing a quorum that corresponds to a threshold number of nodes, wherein the new node is operational at a time when the quorum is formed and is not selected as a member of the quorum.

2. The method of claim 1, wherein the storage pool is presented to a user virtual machine by a controller comprising a service virtual machine running above a hypervisor.

3. The method of claim 1, wherein the storage pool is exposed via internet small computer interface (iSCSI) or network files system (NFS) and mounted as a virtual disk for a user virtual machine.

4. The method of claim 1, wherein the quorum is identified from a subset of nodes equal to a replication factor.

5. The method of claim 1, wherein the new node is maintained in the state that identifies the new node for non-inclusion in a quorum while corresponding metadata is not up to date.

6. The method of claim 1, wherein the new node is immediately after the existing node in the ring structure.

7. The method of claim 1, wherein the threshold of the number of nodes correspond to a majority of the number of nodes, and wherein matching results comprise matching metadata corresponding to the I/O request.

8. The method of claim 1, wherein a background process updates a data value for the new node while the new node is in the state that identifies the new node for non-inclusion in a quorum.

9. The method of claim 1, wherein the new node enters a normal operating state from the state that identifies the new node for non-inclusion in a quorum once a background process completes updating a data value for the new node.

10. A apparatus, comprising:

a plurality of nodes, wherein the plurality of nodes comprise memories for storing instructions and processors which, when executing instructions stored in the memories, performs acts comprising;

maintaining a network virtualization environment comprising a plurality of nodes forming a ring structure, wherein a storage pool is formed from locally attached storage devices from the plurality of nodes;

adding a new node that is adjacent to an existing node in the plurality of nodes forming the ring structure, wherein the new node is placed in a state that identifies the new node for non-inclusion in a quorum;

receiving an I/O request for data in the storage pool stored at the existing node;

and

establishing a quorum that corresponds to a threshold number of nodes, wherein the new node is operational at a time when the quorum is formed and is not selected as a member of the quorum.

11. The apparatus of claim 10, wherein the storage pool is presented to a user virtual machine by a controller comprising a service virtual machine running above a hypervisor.

12. The apparatus of claim 10, wherein the storage pool is exposed via internet small computer interface (iSCSI) or network files system (NFS) and mounted as a virtual disk for a user virtual machine.

13. The apparatus of claim 10, wherein the quorum is identified from a subset of nodes equal to a replication factor.

14. The apparatus of claim 10, wherein the new node is maintained in the state that identifies the new node for non-inclusion in a quorum while corresponding metadata is not up to date.

15. The apparatus of claim 10, wherein the new node is immediately after the existing node in the ring structure.

16. The apparatus of claim 10, wherein the threshold of the number of nodes correspond to a majority of the number of nodes, and wherein matching results comprise matching metadata corresponding to the I/O request.

17. The apparatus of claim 10, wherein a background process updates a data value for the new node while the new node is in the state that identifies the new node for non-inclusion in a quorum.

18. The apparatus of claim 10, wherein the new node enters a normal operating state from the state that identifies the new node for non-inclusion in a quorum once a background process completes updating a data value for the new node.

19. A non-transitory computer readable medium having a sequence of instructions, which when executed by a processor causes a set of acts, comprising:

maintaining a network virtualization environment comprising a plurality of nodes forming a ring structure, wherein a storage pool is formed from locally attached storage devices from the plurality of nodes;

adding a new node that is adjacent to an existing node in the plurality of nodes forming the ring structure, wherein the new node is placed in a state that identifies the new node for non-inclusion in a quorum;

receiving an I/O request for data in the storage pool stored at the existing node;

and

establishing a quorum that corresponds to a threshold number of nodes, wherein the new node is operational at a time when the quorum is formed and is not selected as a member of the quorum.

20. The computer readable medium of claim 19, wherein the storage pool is presented to a user virtual machine by a controller comprising a service virtual machine running above a hypervisor.

21. The computer readable medium of claim 19, wherein the storage pool is exposed via internet small computer interface (iSCSI) or network files system (NFS) and mounted as a virtual disk for a user virtual machine.

22. The computer readable medium of claim 19, wherein the quorum is identified from a subset of nodes equal to a replication factor.

23. The computer readable medium of claim 19, wherein the new node is maintained in the state that identifies the new node for non-inclusion in a quorum while corresponding metadata is not up to date.

24. The computer readable medium of claim 19, wherein the new node is immediately after the existing node in the ring structure.

25. The computer readable medium of claim 19, wherein the threshold of the number of nodes correspond to a majority of the number of nodes, and wherein matching results comprise matching metadata corresponding to the I/O request.

26. The computer readable medium of claim 19, wherein a background process updates a data value for the new node while the new node is in the state that identifies the new node for non-inclusion in a quorum.

27. The computer readable medium of claim 19, wherein the new node enters a normal operating state from the state that identifies the new node for non-inclusion in a quorum once a background process completes updating a data value for the new node.