File deduplication and scan reduction in a virtualization environment

• US 9,424,058 B1
• Filed: 09/23/2013
• Issued: 08/23/2016
• Est. Priority Date: 09/23/2013
• Status: Active Grant

• Current Assignee: CA, Inc. (d/b/a CA Technologies) (Broadcom, Inc.)
• Original Assignee: Symantec Corporation (NortonLifeLock Inc.)
• Inventors: Wasson, Scott G, Klimov, Alex, Vos, Jacob J
• Primary Examiner(s): SWIFT, CHARLES M
• Application Number:
  US14/034,405
• Time in Patent Office: 1,065 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/17 Details of further file system functions
• CPC Class Codes: G06F 16/174 Redundancy elimination performed by the file system management of the data involved in backup or backup restore using de-duplication of the data G06F11/14
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 2009/45562 Creating, deleting, cloning virtual machine instances
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 2009/45575 Starting, stopping, suspending or resuming virtual machine instances
• CPC Class Codes: G06F 2009/45579 I/O management, e.g. providing access to device drivers or storage
• CPC Class Codes: G06F 2009/45583 Memory management, e.g. access or allocation
• CPC Class Codes: G06F 9/455 Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

A virtual machine template is created. The template includes a file system containing files to be deduplicated across multiple virtual machines. For each file to deduplicate, a hash of the content is generated and stored in association with the file. The content of the file is moved from the virtual machine template to a file store. The entry for the file in the store is indexed according to the hash. Multiple virtual machines are created by cloning the template, each containing a copy of its file system and the hashes stored locally in association with the corresponding deduplicated files. File access operations are monitored on each one of the multiple virtual machines, and attempts to access deduplicated file are detected. In response, the corresponding locally stored hash is used to retrieve the content of the file from the central file store, and provide it to the virtual machine.

Claims

1. A computer implemented method for enabling file deduplication and scan reduction across multiple virtual machines in a virtualization environment of a single host computer, the method comprising the steps of:

creating a virtual machine template based upon which to create multiple virtual machines on the host computer, wherein the virtual machine template comprises a description of a virtual machine in a known good state having a file system containing at least some files to deduplicate across the multiple virtual machines;

for each specific file in the file system of the virtual machine template to deduplicate across the multiple virtual machines, deduplicating the specific file by;

generating a hash of content of the specific file;

storing the generated hash locally on the virtual machine template in association with the specific file; and

moving the content of the specific file from the virtual machine template to a central file store residing independently of the virtual machine template and the multiple virtual machines;

creating multiple virtual machines by cloning the virtual machine template, wherein each one of the multiple virtual machines cloned from the virtual machine template contains a copy of the file system of the virtual machine template and a copy of the generated hashes of the content of the deduplicated files, the copy of the hashes being stored locally on the specific virtual machine in association with the corresponding deduplicated files;

monitoring file access operations on each one of the multiple virtual machines cloned from the virtual machine template;

on each one of the multiple virtual machines cloned from the virtual machine template, in response to detecting an attempt to access a deduplicated file the content of which is in the central file store and is not present on the specific virtual machine, using a corresponding hash stored locally on the specific virtual machine in association with the specific file to retrieve the content of the specific file from the central file store;

in response to a specific virtual machine updating a deduplicated file the content of which is in the central file store and is not present on the specific virtual machine;

storing the updated file on the specific virtual machine and not in the central file store; and

deleting the hash stored in association with the updated file from the specific virtual machine; and

in response to a specific virtual machine deleting a deduplicated file the content of which is in the central file store and is not present on the specific virtual machine;

deleting the hash stored in association with the specific file from the specific virtual machine and deleting an entry for the specific file from the file system of the specific virtual machine.

2. The method of claim 1 wherein moving the content of a specific file from the virtual machine template to the central file store further comprises:

indexing an entry of the content of the specific file in the file store according to the hash of the content.

3. The method of claim 1 wherein moving the content of a specific file from the virtual machine template to a central file store further comprises:

removing the content from the virtual machine template by truncating the specific file on the virtual machine template.

4. The method of claim 1 wherein generating a hash of the content of a specific file and storing the generated hash locally on the virtual machine template in association with the specific file further comprises:

storing the generated hash as an extended file attribute of the specific file, in a database of extended file attributes on the virtual machine template.

5. The method of claim 1 further comprising:

after moving the content of each specific file in the file system of the virtual machine template to deduplicate to the central file store, marking all unused sectors of the file system of the virtual machine template as being free.

6. The method of claim 1 wherein detecting an attempt to access a deduplicated file the content of which is in the central file store and is not present on the specific virtual machine further comprises:

detecting an attempt to access a file on the specific virtual machine; and

determining that the content of the specific file resides in the central file store instead of locally on the specific virtual machine in response to a hash of the content of the specific file being present on the specific virtual machine.

7. The method of claim 1 wherein using a corresponding hash stored locally on the specific virtual machine in association with the specific file to retrieve the content of the specific file from the central file store further comprises:

making a request to the central file store to access the content of the specific file, the request comprising the hash of the content of the specific file; and

using the hash to locate and retrieve the content of the specific file in the central file store.

8. The method of claim 7 further comprising:

providing the content of the specific file from the central file store to the specific virtual machine as a stream.

9. The method of claim 7 further comprising:

providing the content of the specific file from the central file store to the specific virtual machine as blocks.

10. The method of claim 1 wherein the central file store residing independently of the virtual machine template and the multiple virtual machines further comprises:

the central file store residing at a level of a hypervisor of the virtualization environment of the single host computer.

11. The method of claim 1 wherein the central file store residing independently of the virtual machine template and the multiple virtual machines further comprises:

the central file store residing on a remote server computer and communicating with the multiple virtual machines on the host computer via a network protocol.

12. The method of claim 1 wherein the central file store residing independently of the virtual machine template and the multiple virtual machines further comprises:

the central file store residing on a virtual appliance on the host computer and communicating with the multiple virtual machines via a hypervisor of the virtualization environment.

13. At least one non-transitory computer readable-storage medium for enabling file deduplication and scan reduction across multiple virtual machines in a virtualization environment of a single host computer, the at least one non-transitory computer readable-storage medium storing computer executable instructions that, when loaded into computer memory and executed by at least one processor of a computing device, cause the computing device to perform the following steps:

creating a virtual machine template based upon which to create multiple virtual machines on the host computer, wherein the virtual machine template comprises a description of a virtual machine in a known good state having a file system containing at least some files to deduplicate across the multiple virtual machines;

for each specific file in the file system of the virtual machine template to deduplicate across the multiple virtual machines, deduplicating the specific file by;

generating a hash of content of the specific file;

storing the generated hash locally on the virtual machine template in association with the specific file; and

moving the content of the specific file from the virtual machine template to a central file store residing independently of the virtual machine template and the multiple virtual machines;

creating multiple virtual machines by cloning the virtual machine template, wherein each one of the multiple virtual machines cloned from the virtual machine template contains a copy of the file system of the virtual machine template and a copy of the generated hashes of the content of the deduplicated files, the copy of the hashes being stored locally on the specific virtual machine in association with the corresponding deduplicated files;

monitoring file access operations on each one of the multiple virtual machines cloned from the virtual machine template;

on each one of the multiple virtual machines cloned from the virtual machine template, in response to detecting an attempt to access a deduplicated file the content of which is in the central file store and is not present on the specific virtual machine, using a corresponding hash stored locally on the specific virtual machine in association with the specific file to retrieve the content of the specific file from the central file store; and

in response to a specific virtual machine updating a deduplicated file the content of which is in the central file store and is not present on the specific virtual machine;

storing the updated file on the specific virtual machine and not in the central file store; and

deleting the hash stored in association with the updated file from the specific virtual machine; and

in response to a specific virtual machine deleting a deduplicated file the content of which is in the central file store and is not present on the specific virtual machine;

deleting the hash stored in association with the specific file from the specific virtual machine and deleting an entry for the specific file from the file system of the specific virtual machine.

14. The at least one non-transitory computer readable-storage medium of claim 13 wherein moving the content of a specific file from the virtual machine template to the central file store further comprises:

indexing an entry of the content of the specific file in the file store according to the hash of the content.

15. The at least one non-transitory computer readable-storage medium of claim 13 wherein generating a hash of the content of a specific file and storing the generated hash locally on the virtual machine template in association with the specific file further comprises:

storing the generated hash as an extended file attribute of the specific file, in a database of extended file attributes on the virtual machine template.

16. A computer implemented method for enabling file deduplication and scan reduction across multiple virtual machines in a virtualization environment of a single host computer, the method comprising the steps of:

creating a virtual machine template based upon which to create multiple virtual machines on the host computer, wherein the virtual machine template comprises a description of a virtual machine in a known good state having a file system containing at least some files to deduplicate across the multiple virtual machines;

for each specific file in the file system of the virtual machine template to deduplicate across the multiple virtual machines, deduplicating the specific file by;

generating a hash of content of the specific file;

storing the generated hash locally on the virtual machine template in association with the specific file; and

moving the content of the specific file from the virtual machine template to a central file store residing independently of the virtual machine template and the multiple virtual machines;

creating multiple virtual machines by cloning the virtual machine template, wherein each one of the multiple virtual machines cloned from the virtual machine template contains a copy of the file system of the virtual machine template and a copy of the generated hashes of the content of the deduplicated files, the copy of the hashes being stored locally on the specific virtual machine in association with the corresponding deduplicated files;

monitoring file access operations on each one of the multiple virtual machines cloned from the virtual machine template;

on each one of the multiple virtual machines cloned from the virtual machine template, in response to detecting an attempt to access a deduplicated file the content of which is in the central file store and is not present on the specific virtual machine, using a corresponding hash stored locally on the specific virtual machine in association with the specific file to retrieve the content of the specific file from the central file store; and

in response to a specific virtual machine updating a deduplicated file the content of which is in the central file store and is not present on the specific virtual machine;

storing a delta from original content to updated content of the deduplicated file in the central file store;

generating a hash of the updated content of the specific file; and

storing the generated hash of the updated content on the specific virtual machine in association with the specific file.

17. The method of claim 16 further comprising:

periodically scanning the central file store and each one of the multiple virtual machines cloned from the virtual machine template;

identifying at least one specific file in the central file store that is no longer in use by any one of the multiple virtual machines cloned from the virtual machine template; and

deleting the at least one specific file from the central file store.

18. The method of claim 16 wherein moving the content of a specific file from the virtual machine template to the central file store further comprises:

indexing an entry of the content of the specific file in the file store according to the hash of the content.

19. The method of claim 16 wherein generating a hash of the content of a specific file and storing the generated hash locally on the virtual machine template in association with the specific file further comprises:

storing the generated hash as an extended file attribute of the specific file, in a database of extended file attributes on the virtual machine template.

20. The method of claim 16 wherein detecting an attempt to access a deduplicated file the content of which is in the central file store and is not present on the specific virtual machine further comprises:

detecting an attempt to access a file on the specific virtual machine; and

determining that the content of the specific file resides in the central file store instead of locally on the specific virtual machine in response to a hash of the content of the specific file being present on the specific virtual machine.