De-duplication in a virtualized storage environment

• US 9,213,494 B1
• Filed: 08/20/2013
• Issued: 12/15/2015
• Est. Priority Date: 09/05/2007
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Yueh, Jedidiah
• Primary Examiner(s): Bradley, Matthew
• Application Number:
  US13/971,740
• Time in Patent Office: 847 Days
• Field of Search: 711/162
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/0683 Plurality of storage devices

Abstract

In one example, a method for de-duplicating redundant data in a virtualized storage environment includes operating a data de-duplication application on a host system that is one of a plurality of host systems in a computer architecture, where the data de-duplication application is operable to globally de-duplicate data in a pooled storage capacity that comprises a virtualization of a plurality of storage devices.

Claims

1. A method for de-duplicating redundant data in a virtualized storage environment, the method comprising:

operating a data de-duplication application on a host system that is one of a plurality of host systems in a computer architecture, wherein the plurality of host systems operate on an abstraction layer that abstracts multiple storage devices into a single pool of storage capacity for the plurality of hosts systems such that the single pool of storage capacity includes a virtualization of a plurality of storage devices and wherein the host system is allocated a portion of the single pool of storage capacity;

presenting the de-duplication application, by the abstraction layer, with a representation of the single pool of storage capacity; and

de-duplicating the single pool of storage capacity, by the data de-duplication application by identifying and removing redundant data such that a single instance of the each data remains in the single pool of storage capacity.

2. The method as recited in claim 1, wherein a data de-duplication operation performed by the data de-duplication application comprises:

identifying first data in the pooled storage capacity that is identical to second data in the pooled storage capacity;

deleting the second data from the pooled storage capacity; and

replacing the second data with a pointer pointing to the identical first data.

3. The method as recited in claim 2, wherein identifying first data in the pooled storage capacity that is identical to second data in the pooled storage capacity comprises:

performing a hash function on each of the first and second data;

comparing the resulting hash values for the first and second data to each other; and

determining that the hash values are identical.

4. The method as recited in claim 2, wherein the pointer pointing to the identical first data comprises the hash value of the first or second data.

5. The method as recited in claim 1, wherein the method further comprises creating the pooled storage capacity by applying the abstraction layer to the plurality of storage devices, the abstraction layer presenting the representation of all the pooled storage capacity to the data de-duplication application.

6. The method as recited in claim 1, wherein the data de-duplication application operates in connection with data written to and read from the pooled storage capacity by one or more of the host systems.

19. The method of claim 1, further comprising de-duplicating the single pool of storage capacity as the host systems as data from the host systems is written to the single pool of storage capacity and/or after the data from the hosts systems is written to the single pool of storage capacity.

7. A physical storage device having stored therein computer-executable instructions stored on a non-transitory computer-readable medium which, when executed by one or more processing devices, perform de-duplication of redundant data, wherein performing de-duplication of redundant data includes:

operating a data de-duplication application on a host system that is one of a plurality of host systems in a computer architecture, wherein the plurality of host systems operate on a virtualization layer that abstracts multiple storage devices into a single pool of storage capacity for the plurality of hosts systems such that the single pool of storage capacity includes a virtualization of a plurality of storage devices and wherein the host system is allocated a portion of the single pool of storage capacity;

presenting the de-duplication application, by the virtualization layer, with a representation of the single pool of storage capacity; and

de-duplicating redundant data from the single pool of storage capacity, by the data de-duplication application, by identifying and removing the redundant data such that a single instance of the each data remains in the single pool of storage capacity for the plurality of host systems.

8. The physical storage device as recited in claim 7, wherein operating the data de-duplication application comprises:

identifying first data in the pooled storage capacity that is identical to second data in the pooled storage capacity;

deleting the second data from the pooled storage capacity; and

replacing the second data with a pointer pointing to the identical first data.

9. The physical storage device as recited in claim 8, wherein identifying first data in the pooled storage capacity that is identical to second data in the pooled storage capacity comprises:

performing a hash function on each of the first and second data;

comparing the resulting hash values for the first and second data to each other; and

determining that the hash values are identical.

10. The physical storage device as recited in claim 8, wherein the pointer pointing to the identical first data comprises the hash value of the first or second data.

11. The physical storage device as recited in claim 7, wherein performing de-duplication of the redundant data further comprises creating the pooled storage capacity by applying an abstraction layer to the plurality of storage devices, the abstraction layer presenting the representation of all the pooled storage capacity to the data de-duplication application, wherein the abstraction layer comprises the virtualization layer.

12. The physical storage device as recited in claim 7, wherein the data de-duplication application operates in connection with data written to and read from the pooled storage capacity by one or more of the host systems.

13. A host system, comprising:

an operating system operable to manage sharing of host system resources among applications running on the host system; and

a physical storage device having stored therein a data de-duplication application running on the host system and including computer-executable instructions which, when executed by one or more processing devices, perform de-duplication of redundant data, wherein performing de-duplication of redundant data includes;

operating the data de-duplication application on the host system, where the host system is one of a plurality of host systems in a computer architecture, wherein the plurality of host systems operate on a virtualization layer that abstracts multiple storage devices into a single pool of storage capacity that is virtualized for the plurality of hosts systems and wherein the host system is allocated a portion of the single pool of storage capacity;

presenting the de-duplication application, by the virtualization layer, with a representation of the single pool of storage capacity; and

de-duplicating the single pool of storage capacity, by the data de-duplication application, by identifying and removing redundant data such that a single instance of the each data remains in the single pool of storage capacity for the plurality of host systems.

14. The host system as recited in claim 13, wherein operating the data de-duplication application comprises:

identifying first data in the pooled storage capacity that is identical to second data in the pooled storage capacity;

deleting the second data from the pooled storage capacity; and

replacing the second data with a pointer pointing to the identical first data.

15. The host system as recited in claim 14, wherein identifying first data in the pooled storage capacity that is identical to second data in the pooled storage capacity comprises:

performing a hash function on each of the first and second data;

comparing the resulting hash values for the first and second data to each other; and

determining that the hash values are identical.

16. The host system as recited in claim 14, wherein the pointer pointing to the identical first data comprises the hash value of the first or second data.

17. The host system as recited in claim 13, wherein performing de-duplication of redundant data further comprises creating the pooled storage capacity by applying the virtualization layer to the plurality of storage devices, the abstraction layer presenting the representation of all the pooled storage capacity to the data de-duplication application.

18. The host system as recited in claim 13, wherein the data de-duplication application operates in connection with data written to and read from the pooled storage capacity by one or more of the host systems.