Method and system for managing de-duplication of data in a virtualization environment

• US 9,389,887 B1
• Filed: 01/18/2013
• Issued: 07/12/2016
• Est. Priority Date: 08/10/2011
• Status: Active Grant

• Current Assignee: Nutanix Incorporated
• Original Assignee: Nutanix Incorporated
• Inventors: Aron, Mohit, Pandey, Dheeraj, Singh, Ajeet
• Primary Examiner(s): Swift, Charles
• Application Number:
  US13/744,649
• Time in Patent Office: 1,271 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1873 Versioning file systems, temporal file systems, e.g. file system supporting different historic versions of files
• CPC Class Codes: G06F 2009/45562 Creating, deleting, cloning virtual machine instances
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 2009/45575 Starting, stopping, suspending or resuming virtual machine instances
• CPC Class Codes: G06F 2009/45579 I/O management, e.g. providing access to device drivers or storage
• CPC Class Codes: G06F 2009/45583 Memory management, e.g. access or allocation
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 9/455 Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/48 Program initiating; Program switching, e.g. by interrupt
• CPC Class Codes: G06F 9/4806 Task transfer initiation or dispatching
• CPC Class Codes: G06F 9/4843 by program, e.g. task dispatcher, supervisor, operating system
• CPC Class Codes: G06F 9/485 Task life-cycle, e.g. stopping, restarting, resuming execution G06F9/4881 takes precedence
• CPC Class Codes: G06F 9/4856 : resumption being on a different machine, e.g. task migration, virtual machine migration G06F9/5088 takes precedence
• CPC Class Codes: G06F 9/4881 : Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/50 : Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06F 9/5005 : to service a request
• CPC Class Codes: G06F 9/5011 : the resources being hardware resources other than CPUs, Servers and Terminals
• CPC Class Codes: G06F 9/5016 : the resource being the memory
• CPC Class Codes: G06F 9/5022 : Mechanisms to release resources
• CPC Class Codes: G06F 9/5027 : the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: G06F 9/5061 : Partitioning or combining of resources
• CPC Class Codes: G06F 9/5072 : Grid computing
• CPC Class Codes: G06F 9/5077 : Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: G06F 9/5083 : Techniques for rebalancing the load in a distributed system
• CPC Class Codes: G06F 9/5088 : involving task migration
• CPC Class Codes: H04L 49/356 : for storage area networks

Abstract

A system for managing de-duplication of data in a virtualization environment, that includes a plurality of nodes, wherein the plurality of nodes implements a virtualization environment, a plurality of primary storage devices that are accessed by virtual machines in the virtualization environment, and wherein data undergoes de-duplication in the virtualization environment such that an instruction to store a data item that is issued by a virtual machine is not duplicatively stored if the data item is already stored in the plurality of the primary storage devices.

Claims

1. A system for managing de-duplication of data in a virtualization environment, comprising:

a plurality of nodes having multiple storage controllers implemented as virtual machines, the multiple storage controllers distributed across multiple nodes, wherein the plurality of nodes implements a virtualization environment;

a plurality of primary storage devices that are accessed by virtual machines in the virtualization environment, wherein the plurality of primary storage devices are managed by the multiple storage controllers; and

wherein the plurality of primary storage devices correspond to a global storage pool having a local storage device on a first node of the plurality of nodes and a local storage device on a second node of the plurality of nodes, the second node being different from the first node, and wherein a virtual disk is exposed to the virtual machines as a set of addressable storage units corresponding to a plurality of virtual disks structured from the global storage pool and data within the set of addressable storage units is organized as extents and the extents are grouped within extent groups, wherein data undergoes de-duplication in the virtualization environment such that an instruction issued by a virtual machine to store a data item does not result in the data item being duplicatively stored if the data item is already stored in the global storage pool, and de-duplication is enabled on an extent group basis.

2. The system of claim 1, in which a new extent is created to store the data item if the data item does not correspond to the extent that already exists.

3. The system of claim 1, in which fingerprinting determines whether the data item is already stored in the plurality of the primary storage devices.

4. The system of claim 1, further comprising a counter that is maintained to correspond to a de-duplicated copy of the data item.

5. The system of claim 4, in which the counter identifies whether or how many entities are relying upon the de-duplicated copy of the data item, wherein each new entity relying on the de-duplicated copy causes an increment to the counter and each entity that no longer relies on the de-duplicated copy causes a decrement to the counter.

6. The system of claim 1, in which a service virtual machine (VM) implements the de-duplication.

7. The system of claim 1, in which the de-duplication is directly performed when storing the data item within the primary storage devices.

8. A method for implementing de-duplication in a virtualization environment, comprising:

receiving an instruction by a virtual machine to store a data item in a virtualization environment, the virtualization environment implemented by a plurality of nodes having multiple storage controllers implemented as virtual machines distributed across the plurality of nodes, the multiple storage controllers managing a plurality of primary storage devices that are accessed by virtual machines in the virtualization environment to store data; and

performing de-duplication on data in the virtualization environment such that an instruction issued by a virtual machine to store a data item to a virtual disk does not result in the data item being duplicatively stored if the data item is already stored in a global storage pool having a local storage device on a first node of the plurality of nodes and a local storage device on a second node of the plurality of nodes, the second node being different from the first node, wherein the virtual disk is exposed to the virtual machines as a set of addressable storage units corresponding to a plurality of virtual disks structured from the global storage pool and data within the set of addressable storage units is organized as extents, and the extents are grouped within extent groups, and de-duplication is enabled on an extent group basis.

9. The method of claim 8, in which a new extent is created to store the data item if the data item does not correspond to the extent that already exists.

10. The method of claim 8, in which fingerprinting is performed on the data item to determine whether the data item is already stored in the plurality of the primary storage devices.

11. The method of claim 8, in which a counter is maintained to correspond to a de-duplicated copy of the data item.

12. The method of claim 11, in which the counter identifies whether or how many entities are relying upon the de-duplicated copy of the data item, wherein each new entity relying on the de-duplicated copy causes an increment to the counter and each entity that no longer relies on the de-duplicated copy causes a decrement to the counter.

13. A computer program product embodied on a non-transitory computer readable medium, the non-transitory computer readable medium having stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a method for implementing de-duplication in a virtualization environment, the method comprising:

receiving an instruction by a virtual machine to store a data item in a virtualization environment, the virtualization environment implemented by a plurality of nodes having multiple storage controllers implemented as virtual machines distributed across the plurality of nodes, the multiple storage controllers managing a plurality of primary storage devices that are accessed by virtual machines in the virtualization environment to store data; and

performing de-duplication on data in the virtualization environment such that an instruction issued by a virtual machine to store a data item to a virtual disk does not result in the data item being duplicatively stored if the data item is already stored in a global storage pool having a local storage device on a first node of the plurality of nodes and a local storage device on a second node of the plurality of nodes, the second node being different from the first node, wherein the virtual disk is exposed to the virtual machines as a set of addressable storage units corresponding to a plurality of virtual disks structured from the global storage pool and data within the set of addressable storage units is organized as extents, and the extents are grouped within extent groups, and de-duplication is enabled on an extent group basis.

14. The computer program product of claim 13, in which a new extent is created to store the data item if the data item does not correspond to the extent that already exists.

15. The computer program product of claim 13, in which fingerprinting is performed on the data item to determine whether the data item is already stored in the plurality of the primary storage devices.

16. The computer program product of claim 13, in which a counter is maintained to correspond to a de-duplicated copy of the data item.

17. The computer program product of claim 16, in which the counter identifies whether or how many entities are relying upon the de-duplicated copy of the data item, wherein each new entity relying on the de-duplicated copy causes an increment to the counter and each entity that no longer relies on the de-duplicated copy causes a decrement to the counter.