De-duplication in a virtualized storage environment

• US 8,549,245 B2
• Filed: 05/21/2012
• Issued: 10/01/2013
• Est. Priority Date: 09/05/2007
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Yueh, Jedidiah
• Primary Examiner(s): BRADLEY, MATTHEW A
• Application Number:
  US13/476,666
• Publication Number:
  US 20120233425A1
• Time in Patent Office: 498 Days
• Field of Search: 711/162
• US Class Current: 711/162
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/0683 Plurality of storage devices

Abstract

A data de-duplication application de-duplicates redundant data in the pooled storage capacity of a virtualized storage environment. The virtualized storage environment includes a plurality of storage devices and a virtualization or abstraction layer that aggregates all or a portion of the storage capacity of each storage device into a single pool of storage capacity, all or portions of which can be allocated to one or more host systems. For each host system, the virtualization layer presents a representation of at least a portion of the pooled storage capacity wherein the corresponding host system can read and write data. The data de-duplication application identifies redundant data in the pooled storage capacity and replaces it with one or more pointers pointing to a single instance of the data. The de-duplication application can operate on fixed or variable size blocks of data and can de-duplicate data either post-process or in-line.

Claims

1. A host system configured to write data to, and read data from, a portion of a pooled storage capacity, and the host system comprising:

a data de-duplication application operable to de-duplicate data in the portion of the pooled storage capacity, wherein a data de-duplication operation performed by the data de-duplication application comprises;

identifying first data in the pooled storage capacity that is identical to second data in the pooled storage capacity;

deleting the second data from the pooled storage capacity; and

replacing the second data with the pointer pointing to the identical first data; and

a virtualization agent that is operable to;

pool storage capacity by virtualizing a plurality of storage devices; and

present a representation of the portion of the pooled storage capacity to the data de-duplication application.

2. The host system of claim 1, wherein the virtualization agent is operable to:

intercept read and write requests from other systems; and

map virtual storage addresses included in the read and write requests to physical storage addresses in the pooled storage capacity.

3. The host system as recited in claim 1, wherein identifying first data in the pooled storage capacity that is identical to second data in the pooled storage capacity includes performing a hash function on each of the first and second data, comparing the resulting hash values for the first and second data to each other, and determining that the hash values are identical.

4. The host system as recited in claim 3, wherein the pointer pointing to the identical first data comprises the hash value of the first or second data.

5. The host system of claim 1, wherein the data de-duplication application is operable to de-duplicate data written from one or more other systems to the pooled storage capacity either post-process or in-line.

6. The host system as recited in claim 1, wherein the host system comprises a server.

7. The host system as recited in claim 6, wherein the server further includes at least one of a file server application, or an email server application.

8. A computer architecture, comprising:

a host system configured to write data to, and read data from, a portion of a pooled storage capacity, and the host system comprising;

a data de-duplication application operable to de-duplicate data in the portion of the pooled storage capacity, wherein a data de-duplication operation performed by the data de-duplication application comprises;

identifying first data in the pooled storage capacity that is identical to second data in the pooled storage capacity;

deleting the second data from the pooled storage capacity; and

replacing the second data with the pointer pointing to the identical first data;

a virtualization layer that is operable to;

pool storage capacity by virtualizing a plurality of storage devices; and

present a representation of the portion of the pooled storage capacity to the data de-duplication application; and

one of;

a storage platform configured to communicate with a plurality of host systems including the host system, wherein the storage platform provides the virtualization layer;

or

a switch configured to communicate with a plurality of host systems including the host system, wherein the switch provides the virtualization layer.

9. The computer architecture as recited in claim 8, wherein identifying first data in the pooled storage capacity that is identical to second data in the pooled storage capacity includes performing a hash function on each of the first and second data, comparing the resulting hash values for the first and second data to each other, and determining that the hash values are identical.

10. The computer architecture as recited in claim 9, wherein the pointer pointing to the identical first data comprises the hash value of the first or second data.

11. The computer architecture of claim 8, wherein the data de-duplication application is operable to de-duplicate data written from one or more other systems to the pooled storage capacity either post-process or in-line.

12. The computer architecture as recited in claim 8, further comprising a plurality of storage devices configured to be virtualized as a pooled storage capacity by the virtualization layer.

13. The computer architecture as recited in claim 8 wherein if the computer architecture includes the storage platform, the computer architecture further comprises a switch by way of which the storage platform is able to communicate with the plurality of host systems.

14. The computer architecture as recited in claim 8, wherein the switch comprises an in-band virtualization appliance.

15. A computer architecture, comprising:

a host system configured to write data to, and read data from, a portion of a pooled storage capacity, and the host system comprising;

a data de-duplication application operable to de-duplicate data in the portion of the pooled storage capacity, wherein a data de-duplication performed by operation of the data de-duplication application comprises;

identifying first data in the pooled storage capacity that is identical to second data in the pooled storage capacity;

deleting the second data from the pooled storage capacity; and

replacing the second data with the pointer pointing to the identical first data;

a thin virtualization driver that is operable to;

pool storage capacity by virtualizing a plurality of storage devices; and

present a representation of the portion of the pooled storage capacity to the data de-duplication application; and

an out-of-band appliance operable to provide storage mappings to the thin virtualization driver.

16. The host computer architecture as recited in claim 15, wherein identifying first data in the pooled storage capacity that is identical to second data in the pooled storage capacity includes performing a hash function on each of the first and second data, comparing the resulting hash values for the first and second data to each other, and determining that the hash values are identical.

17. The computer architecture as recited in claim 15, wherein the pointer pointing to the identical first data comprises the hash value of the first or second data.