De-duplication in a virtualized server environment

• US 8,880,797 B2
• Filed: 09/28/2007
• Issued: 11/04/2014
• Est. Priority Date: 09/05/2007
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Yueh, Jedidiah
• Primary Examiner(s): Rones, Charles
• Assistant Examiner(s): DARE, RYAN A
• Application Number:
  US11/864,756
• Publication Number:
  US 20090063528A1
• Time in Patent Office: 2,594 Days
• US Class Current: 711/113
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques

Abstract

A data de-duplication application de-duplicates redundant data on the primary storage read/write pathway of a virtualized server environment. The virtualized server environment comprises one or more server applications operating on a virtualization layer provided on a computer architecture that includes memory (e.g., RAM, cache memory) for temporarily storing data and storage (e.g., disk storage) for persistently storing data. The one or more server applications use the read-write pathway to read data into memory from storage and to write data to storage from memory. The de-duplication application identifies redundant data in memory, storage, or both, and replaces the redundant data with one or more pointers pointing to a single copy of the data. The de-duplication application can operate on fixed or variable size blocks of data and can de-duplicate data either post-process or in-line.

Claims

1. A method for de-duplicating redundant data in memory, storage, or both, comprising:

providing a virtual machine including a virtualization layer on a computer architecture comprising memory for temporarily storing data, and storage for persistently storing data, the virtualization layer presenting a representation of at least a portion of the memory and at least a portion of the storage to a server application;

running the server application in the virtual machine on the virtualization layer, the server application reading data from the portion of the storage into the portion of the memory and writing data to the portion of the storage from the portion of the memory through the virtualization layer; and

running a data de-duplication application on the computer architecture to de-duplicate data in the memory and in the storage of the computer architecture, wherein the data de-duplication application treats the memory and the storage as separate systems, and wherein the de-duplication application eliminates redundant data in the memory relative only to the data stored in the memory and eliminates redundant data in the storage relative only to the data stored in the storage.

2. The method of claim 1, wherein the memory comprises one or more of processor registers, cache memory, and random access memory and the storage comprises disk storage.

3. The method of claim 1, wherein de-duplicating data in memory comprises:

identifying first data in memory that is identical to second data in memory;

deleting the second data from memory; and

replacing the second data with a pointer pointing to the identical first data.

4. The method of claim 3, wherein identifying first data in memory that is identical to second data in memory includes performing a hash function on each of the first and second data, comparing the resulting hash values for the first and second data to each other, and determining that the hash values are identical.

5. The method of claim 4, wherein the pointer pointing to the identical first data comprises the hash value of the second data.

6. The method of claim 1, wherein de-duplicating data in storage comprises:

identifying first data in storage that is identical to second data in storage;

deleting the second data from storage; and

replacing the second data with a pointer pointing to the identical first data.

7. The method of claim 6, wherein identifying first data in storage that is identical to second data in storage includes performing a hash function on each of the first and second data, comparing the resulting hash values for the first and second data to each other, and determining that the hash values are identical.

8. The method of claim 7, wherein the pointer pointing to the identical first data comprises the hash value of the second data.

9. The method of claim 1, wherein the server application comprises a plurality of server applications, the virtualization layer presenting a representation of a separate portion of the memory and a separate portion of the storage to each of the server applications.

10. The method of claim 1, wherein running the data de-duplication application on the computer architecture comprises running the data de-duplication application on the virtualization layer.

11. The method of claim 1, wherein:

the data de-duplication application de-duplicates data in memory either as the data is read into memory, or after the data has been read into memory; and

the data de-duplication application de-duplicates data in storage either as the data is written to storage, or after the data has been written to storage.

12. The method of claim 1, wherein de-duplicating data in one or more of memory and storage comprises breaking the data into blocks, the size of the blocks being either fixed or variable.

13. A method for de-duplicating data in a computer environment including one or more virtualized server applications, the method comprising:

providing a virtualization layer on a computer architecture that includes memory for temporarily storing data, and storage for persistently storing data;

operating at least one server application in a virtual machine on the virtualization layer, the at least one server application configured to read data from storage into virtualized memory and to write data from memory to virtualized storage during operation through the virtualization layer which presents a virtualized representation of the computer architecture to the at least one server application, the virtualized representation including the virtualized memory and the virtualized storage;

operating a data de-duplication application on the computer architecture to perform ;

identifying data read into the virtualized memory that is identical to data already stored in memory; and

identifying data written to the virtualized storage that is identical to data already stored in storage; and

replacing ;

identical data read into the virtualized memory with one or more pointers pointing to the corresponding data already stored in memory of the computer architecture; and

identical data written to the virtualized storage with one or more pointers pointing to the corresponding data already stored in storage of the computer architecture,

wherein the data de-duplication of the memory is performed relative to the memory and the data de-duplication of the storage is performed relative to the storage, wherein the de-duplication application eliminates redundant data in the memory relative only to the data stored in the memory and eliminates redundant data in the storage relative only to the data stored in the storage, wherein the data de-duplication application is configured to de-duplicate the data across a plurality of virtual machines including the virtual machine.

14. The method of claim 13, wherein providing a virtualization layer on a computer architecture further comprises pooling resources of the computer architecture.

15. The method of claim 13, wherein the de-duplication application is encapsulated within a virtual machine running on the virtualization layer and wherein the virtualization layer presents a virtualized representation of the computer architecture to an operating system of the virtual machine.

16. The method of claim 13, wherein the computer architecture comprises one or more networks, one or more network devices, or one or more networks and one or more network devices and wherein virtualization layer comprises a virtual infrastructure that connects resources in the computer architecture to users and wherein the virtual infrastructure provides a dynamic mapping of the resources to a plurality of virtual machines.

17. The method of claim 16 wherein the at least one server application comprises a plurality of server applications, each server application being encapsulated within a different one of the plurality of virtual machines and wherein each of the plurality of virtual machines runs an operating system and wherein at least two of the operating systems are identical.

18. The method of claim 17, wherein operating the plurality of server applications includes reading the at least two identical operating systems from storage into memory, and wherein replacing identical data read into memory with one or more pointers pointing to the corresponding data already stored in memory comprises replacing at least one copy of the at least two identical operating systems read into memory with pointers to a single remaining copy of the at least two identical operating systems.

19. The method of claim 13, wherein identical data is replaced with one or more pointers either post-process or in-line.

20. The method of claim 13, further comprising accessing identical data no longer stored in memory, or storage, by using the one or more pointers to access the corresponding data already stored in memory, or storage, respectively.