Virtualized file server smart data ingestion

• US 11,579,861 B2
• Filed: 02/01/2017
• Issued: 02/14/2023
• Est. Priority Date: 02/12/2016
• Status: Active Grant

• Current Assignee: Nutanix Incorporated
• Original Assignee: Nutanix Incorporated
• Inventors: Sharpe, Richard James, Bafna, Kalpesh Ashok, Rathi, Shyamsunder Prayagchand, Deshmukh, Satyajit Sanjeev, Arikatla, Durga Mahesh
• Primary Examiner(s): Nguyen, Thu N
• Application Number:
  US15/422,262
• Publication Number:
  US 20170235762A1
• Time in Patent Office: 2,204 Days
• Field of Search: None
• CPC Class Codes: G06F 11/0712 in a virtual computing platform, e.g. logically partitioned systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1484 involving virtual machines
• CPC Class Codes: G06F 11/201 between storage system components
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/2058 using more than 2 mirrored copies
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2092 Techniques of failing over between control units
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/301 where the computing system is a virtual computing platform, e.g. logically partitioned systems virtual machines G06F9/45533; logical partitioning of resources G06F9/5077
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 16/1727 Details of free space management performed by the file system saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064
• CPC Class Codes: G06F 16/176 : Support for shared access to files; File sharing support
• CPC Class Codes: G06F 16/1774 : Locking methods, e.g. locking methods for file systems allowing shared and concurrent access to files
• CPC Class Codes: G06F 16/183 : Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/1844 : Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/2365 : Ensuring data consistency and integrity
• CPC Class Codes: G06F 2009/45579 : I/O management, e.g. providing access to device drivers or storage
• CPC Class Codes: G06F 2009/45583 : Memory management, e.g. access or allocation
• CPC Class Codes: G06F 2009/45587 : Isolation or security of virtual machine instances
• CPC Class Codes: G06F 2009/45595 : Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 21/53 : by executing in a restricted environment, e.g. sandbox or secure virtual machine
• CPC Class Codes: G06F 21/56 : Computer malware detection or handling, e.g. anti-virus arrangements
• CPC Class Codes: G06F 21/6218 : to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 21/78 : to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 2201/80 : Database-specific techniques
• CPC Class Codes: G06F 2201/815 : Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 2201/825 : the problem or solution involving locking
• CPC Class Codes: G06F 2201/84 : Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 2201/87 : Monitoring of transactions
• CPC Class Codes: G06F 2221/034 : Test or assess a computer or a system
• CPC Class Codes: G06F 2221/2101 : Auditing as a secondary aspect
• CPC Class Codes: G06F 2221/2141 : Access rights, e.g. capability lists, access control lists, access tables, access matrices
• CPC Class Codes: G06F 2221/2145 : Inheriting rights or properties, e.g., propagation of permissions or restrictions within a hierarchy
• CPC Class Codes: G06F 3/0619 : in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0647 : Migration mechanisms
• CPC Class Codes: G06F 3/065 : Replication mechanisms
• CPC Class Codes: G06F 3/0664 : at device level, e.g. emulation of a storage device or system
• CPC Class Codes: G06F 3/0665 : at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/0683 : Plurality of storage devices
• CPC Class Codes: G06F 3/0689 : Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: G06F 8/65 : Updates security arrangements therefor G06F21/57
• CPC Class Codes: G06F 9/4406 : Loading of operating system
• CPC Class Codes: G06F 9/45558 : Hypervisor-specific management and integration aspects
• CPC Class Codes: H04L 41/06 : Management of faults, events, alarms or notifications
• CPC Class Codes: H04L 41/082 : the condition being updates or upgrades of network functionality
• CPC Class Codes: H04L 41/0859 : by keeping history of different configuration generations or by rolling back to previous configuration versions
• CPC Class Codes: H04L 41/5009 : Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
• CPC Class Codes: H04L 67/10 : in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/34 : involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416

Abstract

In one embodiment, a system for managing a virtualization environment includes a set of host machines, each of which includes a hypervisor, virtual machines, and a virtual machine controller, and a data migration system configured to identify one or more existing storage items stored at one or more existing File Server Virtual Machines (FSVMs) of an existing virtualized file server (VFS). For each of the existing storage items, the data migration system is configured to identify a new FSVMs of a new VFS based on the existing FSVM, send a representation of the storage item from the existing FSVM to the new FSVM, such that representations of storage items are sent between different pairs of FSVMs in parallel, and store a new storage item at the new FSVM, such that the new storage item is based on the representation of the existing storage item received by the new FSVM.

Claims

1. A data migration system comprising:

a processor; and

computer readable media encoded with instructions which, when executed, cause the data migration system to;

receive a request to migrate a plurality of storage items from an existing virtualized file server (VFS) configured to process input/output (I/O) requests for the plurality of storage items to a new VFS;

access a mapping of a first set of file server virtual machines of the existing VFS to a second set of file server virtual machines of the new VFS;

identify, at least in part using the mapping, a first destination file server virtual machine of the second set of file server virtual machines to process requests for I/O transactions for a first storage item of the plurality of storage items;

identify, at least in part using the mapping, a second destination file server virtual machine of the second set of file server virtual machines to process requests for I/O transactions for a second storage item of the plurality of storage items;

migrate the first storage item from the existing VFS to the first destination file server virtual machine of the new VFS; and

migrate, at least partially in parallel to migrating the first storage item from the existing VFS to the new VFS, the second storage item from the existing VFS to the second destination file server virtual machine of the new VFS.

2. The system of claim 1, wherein the first storage item is associated with a first existing storage path, and the second storage item is associated with a second existing storage path.

3. The system of claim 2, wherein the first existing storage path and the second existing storage path are retrieved from a sharding map of the existing VFS.

4. The system of claim 1, wherein the data migration system is further configured to identify the first destination file server virtual machine and the second destination file server virtual machine at least in part by executing a sharding algorithm.

5. The system of claim 1, wherein the data migration system is further configured to identify the first destination file server virtual machine and the second destination file server virtual machine at least in part using a sharding map,

wherein a first entry in the sharding map associates the first storage item with a first source file server virtual machine of the first set of file server virtual machines, and

a second entry in the sharding map associates the second storage item with a second source file server virtual machine of the first set of file server virtual machines.

6. The system of claim 1, wherein the data migration system is further configured to send a representation of one of the first and second storage items without waiting for completion of a previous send operation.

19. The system of claim 1, wherein the first set of FSVMs is configured to form a cluster.

21. The system of claim 1, wherein:

the instructions to migrate the first storage item from the existing VFS to the new VFS comprise instructions to migrate the first storage item to a first host hosting the first destination file server virtual machine of the new VFS, and

the instructions to migrate the second storage item from the existing VFS to the new VFS comprise instructions to migrate the second storage item to a second host hosting the second destination file server virtual machine of the new VFS.

22. The system of claim 1, wherein:

the instructions to migrate the first storage item from the existing VFS to the new VFS comprise instructions to migrate the first storage item from a first host of the existing VFS to a first host hosting the first destination file server virtual machine of the new VFS, and

the instructions to migrate the second storage item from the existing VFS to the new VFS comprise instructions to migrate the second storage item from a second host of the existing VFS to a second host hosting the second destination file server virtual machine of the new VFS.

7. One or more computer-readable non-transitory storage media, the media comprising instructions for performing operations comprising:

receiving a request to migrate a first storage item and a second storage item of a plurality of storage items from an existing virtualized file server (VFS) configured to process input/output (I/O) requests for the plurality of storage items to a new VFS;

accessing a mapping of a first set of file server virtual machines of the existing VFS to a second set of file server virtual machines of the new VFS;

identifying at least in part, using the mapping, a first destination file server virtual machine of the second set of file server virtual machines to process requests for I/O transactions for the first storage item;

identifying, at least in part using the mapping, a second destination file server virtual machine of the second set of file server virtual machines to process requests for I/O transactions for the second storage item;

migrating the first storage item from the existing VFS to the first destination file server virtual machine of the new VFS; and

migrating, at least partially in parallel to migrating the first storage item from the existing VFS to the new VFS, the second storage item from the existing VFS to the second destination file server virtual machine of the new VFS.

8. The media of claim 7, wherein the first storage item is associated with a first existing storage path, and the second storage item is associated with a second existing storage path.

9. The media of claim 8, wherein the first existing storage path and the second existing storage path are retrieved from a sharding map of the existing VFS.

10. The media of claim 7, wherein the instructions for identifying the first destination file server virtual machine and the second destination file server virtual machine comprise instructions for executing a sharding algorithm.

11. The media of claim 7, wherein the instructions for identifying the first destination file server virtual machine and the second destination file server virtual machine comprise instructions for accessing a sharding map,

wherein a first entry in the sharding map associates the first storage item with a first source file server virtual machine of the first set of file server virtual machines, and

a second entry in the sharding map associates the second storage item with a second source file server virtual machine of the first set of file server virtual machines.

12. The media of claim 7, wherein the operations further include sending a representation of one of the first and second storage items without waiting for completion of a previous send operation.

20. The media of claim 7, wherein the first set of file server virtual machines comprises a cluster of FSVMs.

23. The media of claim 7, wherein:

migrating the first storage item from the existing VFS to the new VFS comprises migrating the first storage item to a first host hosting the first destination file server virtual machine of the new VFS, and

migrating the second storage item from the existing VFS to the new VFS comprises migrating the second storage item to a second host hosting the second destination file server virtual machine of the new VFS.

24. The media of claim 7, wherein:

migrating the first storage item from the existing VFS to the new VFS comprises migrating the first storage item from a first host of the existing VFS to a first host hosting the first destination file server virtual machine of the new VFS, and

migrating the second storage item from the existing VFS to the new VFS comprises migrating the second storage item from a second host of the existing VFS to a second host hosting the second destination file server virtual machine of the new VFS.

13. A method comprising:

receiving a request to migrate a first storage item and a second storage item of a plurality of storage items from an existing virtualized file server (VFS) configured to process input/output (I/O) requests for the plurality of storage items to a new VFS;

accessing a mapping of a first set of file server virtual machines of the existing VFS to a second set of file server virtual machines of the new VFS;

identifying, at least in part using the mapping, a first destination file server virtual machine of the second set of file server virtual machines to process requests for I/O transactions for the first storage item;

identifying, at least in part using the mapping, a second destination file server virtual machine of the second set of file server virtual machines to process requests for I/O transactions for the second storage item;

migrating the first storage item from the existing VFS to the first destination file server virtual machine of the new VFS; and

migrating, at least partially in parallel to migrating the first storage item from the existing VFS to the new VFS, the second storage item from the existing VFS to the second destination file server virtual machine of the new VFS.

14. The method of claim 13, wherein the first storage item is associated with a first existing storage path, and the second storage item is associated with a second existing storage path.

15. The method of claim 14, wherein the first existing storage path and the second storage path are retrieved from a sharding map of the existing VFS.

16. The method of claim 13, wherein identifying the first destination file server virtual machine and the second destination file server virtual machine comprises executing a sharding algorithm.

17. The method of claim 13, wherein identifying the first destination file server virtual machine and the second destination file server virtual machine comprises accessing a sharding map,

wherein a first entry in the sharding mapping associates the first storage items with a first source file server virtual machine of the first set of file server virtual machines, and

a second entry in the sharding mapping associates the second storage item with a second source file server virtual machine of the first set of file server virtual machines.

18. The method of claim 13 further comprising performing a send operation to send a representation of one of the first or second storage items without waiting for completion of a previous send operation.

25. The method of claim 13, wherein:

migrating the first storage item from the existing VFS to the new VFS comprises migrating the first storage item to a first host hosting the first destination file server virtual machine of the new VFS, and

migrating the second storage item from the existing VFS to the new VFS comprises migrating the second storage item to a second host hosting the second destination file server virtual machine of the new VFS.

26. The method of claim 13, wherein:

migrating the first storage item from the existing VFS to the new VFS comprises migrating the first storage item from a first host of the existing VFS to a first host hosting the first destination file server virtual machine of the new VFS, and

migrating the second storage item from the existing VFS to the new VFS comprises migrating the second storage item from a second host of the existing VFS to a second host hosting the second destination file server virtual machine of the new VFS.