Automated generation of cloned production environments

• US 10,114,565 B2
• Filed: 02/20/2015
• Issued: 10/30/2018
• Est. Priority Date: 11/04/2014
• Status: Active Grant

• Current Assignee: Rubrik, Inc.
• Original Assignee: Rubrik, Inc.
• Inventors: Nithrakashyap, Arvind, To, Kenny, Luo, Jiangbin, Pathak, Sharvanath, Derryberry, Jonathan
• Primary Examiner(s): An, Meng Ai T
• Assistant Examiner(s): Huaracha, Willy W
• Application Number:
  US14/628,033
• Publication Number:
  US 20160124764A1
• Time in Patent Office: 1,348 Days
• Field of Search: 718 1
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/1461 Backup scheduling policy
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1484 involving virtual machines
• CPC Class Codes: G06F 11/202 where processing functionality is redundant redundant communication control functionality G06F11/2005, redundant storage control functionality G06F11/2089
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/84 Mapping; Conversion
• CPC Class Codes: G06F 2009/45562 Creating, deleting, cloning virtual machine instances
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 2009/45579 I/O management, e.g. providing access to device drivers or storage
• CPC Class Codes: G06F 2009/45583 Memory management, e.g. access or allocation
• CPC Class Codes: G06F 2201/80 : Database-specific techniques
• CPC Class Codes: G06F 2201/815 : Virtual
• CPC Class Codes: G06F 2201/84 : Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/0619 : in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0641 : De-duplication techniques
• CPC Class Codes: G06F 3/065 : Replication mechanisms
• CPC Class Codes: G06F 3/0665 : at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0685 : Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: G06F 9/45558 : Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/5077 : Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: H04L 43/0817 : by checking functioning
• CPC Class Codes: H04L 61/5007 : Internet protocol [IP] addresses
• CPC Class Codes: H04L 61/5061 : Pools of addresses
• CPC Class Codes: H04L 67/10 : in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 9/3242 : involving keyed hash functions, e.g. message authentication codes [MACs], CBC-MAC or HMAC
• CPC Class Codes: H04L 9/3247 : involving digital signatures

Abstract

Methods and systems for managing, storing, and serving data within a virtualized environment are described. In some embodiments, a data management system may manage the extraction and storage of virtual machine snapshots, provide near instantaneous restoration of a virtual machine or one or more files located on the virtual machine, and enable secondary workloads to directly use the data management system as a primary storage target to read or modify past versions of data. The data management system may allow a virtual machine snapshot of a virtual machine stored within the system to be directly mounted to enable substantially instantaneous virtual machine recovery of the virtual machine.

Claims

1. A method for generating a cloned virtual machine environment, comprising:

identifying an application running on a first virtual machine at a particular point in time;

determining a set of dependent applications that the application depends on for operation;

determining an ordering of the set of dependent applications using a dependency mapping table such that every application that a particular application of the set of dependent applications depends on precedes the particular application in the ordering;

determining a set of virtual machines that ran the set of dependent applications at the particular point in time;

acquiring a set of snapshots of the set of virtual machines, the set of snapshots corresponds with states of the set of virtual machines at the particular point in time;

generating a second set of virtual machines using the set of snapshots, each virtual machine of the second set of virtual machines is generated in an order that satisfies the ordering of the set of dependent applications;

identifying a first full image associated with the first virtual machine;

generating a first signature using a first full image of the first virtual machine;

identifying a third virtual machine based on a comparison of the first signature associated with the first virtual machine and a third signature associated with the third virtual machine;

identifying one or more incremental files corresponding with one or more snapshots of the first virtual machine; and

generating a second virtual machine corresponding with the first virtual machine subsequent to generating the second set of virtual machines, the generating the second virtual machine includes acquiring the one or more incremental files corresponding with the one or more snapshots of the first virtual machine from a second storage device with a second read speed in parallel with acquiring a third full image of the third virtual machine different from the first virtual machine from a first storage device with a first read speed less than the second read speed, the generating the second virtual machine includes generating the second virtual machine using the third full image of the third virtual machine different from the first virtual machine and the one or more incremental files corresponding with the one or more snapshots of the first virtual machine.

2. The method of claim 1, wherein:

the first storage device comprises a magnetic storage device;

the second storage device comprises a solid-state storage device;

the second virtual machine comprises a cloned version of the first virtual machine; and

the second set of virtual machines comprises cloned versions of the set of virtual machines.

3. The method of claim 1, wherein:

the first virtual machine corresponds with a first virtual machine configuration file;

the third virtual machine corresponds with a third virtual machine configuration file different from the first virtual machine configuration file;

the application depends on a second application of the set of dependent applications; and

the generating the second set of virtual machines includes generating a fourth virtual machine that ran the second application at the particular point in time prior to generating the second virtual machine.

4. The method of claim 1, wherein:

the generating the first signature associated with the first virtual machine includes generating a first plurality of hash values corresponding with a first plurality of data blocks within a first snapshot of the first virtual machine, the first plurality of data blocks within the first snapshot is arranged such that two or more data blocks of a first set of the first plurality of data blocks are spaced at a fixed distance from each other while two or more data blocks of a second set of the first plurality of data blocks are positioned at increasingly greater distances from each other, the first set of the first plurality of data blocks corresponds with a first portion of the first snapshot and the second set of the first plurality of data blocks corresponds with a second portion of the first snapshot that does not overlap with the first portion of the first snapshot.

5. The method of claim 1, wherein:

the generating the second set of virtual machines includes bringing up the second set of virtual machines using the set of snapshots, each virtual machine of the second set of virtual machines is brought up in the order that satisfies the ordering of the set of dependent applications; and

the generating the second virtual machine includes bringing up the second virtual machine subsequent to bringing up the second set of virtual machines.

6. The method of claim 5, wherein:

the bringing up the second virtual machine includes booting up the second virtual machine and running a cloned version of the application using the second virtual machine subsequent to booting up the second virtual machine.

7. The method of claim 5, wherein:

the bringing up the second set of virtual machines includes booting up the second set of virtual machines and executing cloned versions of the set of dependent applications using the second set of virtual machines; and

the bringing up the second virtual machine includes initiating execution of a cloned version of the application using the second virtual machine subsequent to the executing cloned versions of the set of dependent applications using the second set of virtual machines.

8. The method of claim 5, further comprising:

identifying a set of IP addresses used by the first virtual machine and the set of virtual machines; and

configuring the second virtual machine and the second set of virtual machines with the set of IP addresses, the bringing up the second set of virtual machines includes bringing up the second set of virtual machines within a private network, the bringing up the second virtual machine includes bringing up the second virtual machine within the private network.

9. The method of claim 8, further comprising:

configuring a gateway virtual machine to act as a gateway between the private network and a second network outside of the private network.

10. The method of claim 9, wherein:

the configuring a gateway virtual machine includes configuring the gateway virtual machine such that all requests to IP addresses that are not within the set of IP addresses are routed through the gateway virtual machine to the second network; and

the configuring a gateway virtual machine includes configuring the gateway virtual machine such that all requests to IP addresses that are within the set of IP addresses are not routed outside of the private network.

11. The method of claim 1, wherein:

the determining the ordering of the set of dependent applications includes acquiring the dependency mapping table for the set of dependent applications and determining the ordering based on the dependency mapping table, the dependency mapping table includes dependency information for each application of the set of dependent applications.

12. A system for generating a cloned virtual machine environment, comprising:

a memory configured to store a dependency mapping table that includes dependency information of applications; and

one or more processors configured to;

identify an application running on a first virtual machine at a particular point in time;

determine a set of dependent applications that the application depends on for operation;

determine an ordering of the set of dependent applications using the dependency mapping table, the ordering of the set of dependent applications is arranged such that every application that a particular application of the set of dependent applications depends on precedes the particular application in the ordering;

determine a set of virtual machines that ran the set of dependent applications at the particular point in time and acquire a set of snapshots of the set of virtual machines, the set of snapshots corresponds with states of the set of virtual machines at the particular point in time;

generate a second set of virtual machines using the set of snapshots, each virtual machine of the second set of virtual machines is generated in an order that satisfies the ordering of the set of dependent applications;

identify a first full image associated with the first virtual machine;

generate a first signature using the first full image associated with the first virtual machine;

identify a third virtual machine based on a comparison of the first signature associated with the first virtual machine and a third signature associated with the third virtual machine;

identify one or more incremental files corresponding with one or more snapshots of the first virtual machine; and

generate a second virtual machine corresponding with the first virtual machine subsequent to generating the second set of virtual machines, the one or more processors configured to acquire the one or more incremental files corresponding with the one or more snapshots of the first virtual machine from a second storage device with a second read speed in parallel with acquiring a third full image of the third virtual machine different from the first virtual machine from a first storage device with a first read speed less than the second read speed, the one or more processors configured to generate the second virtual machine using the third full image of the third virtual machine different from the first virtual machine and the one or more incremental files corresponding with the one or more snapshots of the first virtual machine.

13. The system of claim 12, wherein:

the one or more processors configured to generate the first signature associated with the first virtual machine via generation of a first plurality of hash values corresponding with a first plurality of data blocks within a first snapshot of the first virtual machine, the first plurality of data blocks within the first snapshot is arranged such that two or more data blocks of a first set of the first plurality of data blocks are spaced at a fixed distance from each other while two or more data blocks of a second set of the first plurality of data blocks are positioned at increasingly greater distances from each other, the first set of the first plurality of data blocks corresponds with a first portion of the first snapshot and the second set of the first plurality of data blocks corresponds with a second portion of the first snapshot that does not overlap with the first portion of the first snapshot;

the second virtual machine comprises a cloned version of the first virtual machine; and

the second set of virtual machines comprises cloned versions of the set of virtual machines.

14. The system of claim 12, wherein:

the first virtual machine corresponds with a first virtual machine configuration file; and

the third virtual machine corresponds with a third virtual machine configuration file different from the first virtual machine configuration file.

15. The system of claim 12, wherein:

the one or more processors configured to generate the second set of virtual machines using the set of snapshots, each virtual machine of the second set of virtual machines is generated in the order that satisfies the ordering of the set of dependent applications; and

the one or more processors configured to generate the second virtual machine subsequent to generating the second set of virtual machines.

16. The system of claim 12, wherein:

the one or more processors configured to boot up the second virtual machine and run a cloned version of the application using the second virtual machine subsequent to booting up the second virtual machine.

17. The system of claim 12, wherein:

the one or more processors configured to boot up the second set of virtual machines and execute cloned versions of the set of dependent applications using the second set of virtual machines, the one or more processors configured to initiate execution of a cloned version of the application using the second virtual machine subsequent to executing the cloned versions of the set of dependent applications using the second set of virtual machines.

18. The system of claim 12, wherein:

the one or more processors configured to identify a set of IP addresses used by the first virtual machine and the set of virtual machines, the one or more processors configured to configure the second virtual machine and the second set of virtual machines with the set of IP addresses, the one or more processors configured to bring up the second set of virtual machines within a private network and bring up the second virtual machine within the private network.

19. The system of claim 18, wherein:

the one or more processors configured to configure a gateway virtual machine to act as a gateway between the private network and a second network outside of the private network.

20. One or more storage devices containing processor readable code for programming one or more processors to perform a method for generating a cloned virtual machine environment, the processor readable code comprising:

processor readable code configured to identify an application running on a first virtual machine at a particular point in time;

processor readable code configured to determine a set of dependent applications that the application depends on for operation;

processor readable code configured to determine an ordering of the set of dependent applications using a dependency mapping table such that every application that a particular application of the set of dependent applications depends on precedes the particular application in the ordering;

processor readable code configured to determine a set of virtual machines that ran the set of dependent applications at the particular point in time;

processor readable code configured to acquire a set of snapshots of the set of virtual machines, the set of snapshots corresponds with states of the set of virtual machines at the particular point in time;

processor readable code configured to generate a second set of virtual machines using the set of snapshots, each virtual machine of the second set of virtual machines is brought up in an order that satisfies the ordering of the set of dependent applications;

processor readable code configured to identify a first full image associated with the first virtual machine;

processor readable code configured to generate a first signature using a first full image of the first virtual machine;

processor readable code configured to identify a third virtual machine based on a comparison of the first signature associated with the first virtual machine and a third signature associated with the third virtual machine;

processor readable code configured to identify one or more incremental files corresponding with one or more snapshots of the first virtual machine;

processor readable code configured to acquire the one or more incremental files corresponding with the one or more snapshots of the first virtual machine from a second storage device comprising a solid-state storage device with a second read speed in parallel with acquiring a third full image of the third virtual machine different from the first virtual machine from a first storage device comprising a magnetic storage device with a first read speed less than the second read speed; and

processor readable code configured to generate a second virtual machine corresponding with the first virtual machine subsequent to bringing up the second set of virtual machines, the generating the second virtual machine includes generating the second virtual machine using the third full image of the third virtual machine different from the first virtual machine and the one or more incremental files corresponding with the one or more snapshots of the first virtual machine.