SYSTEM AND METHOD FOR APPLICATION AWARE DE-DUPLICATION OF DATA BLOCKS ON A VIRTUALIZED STORAGE ARRAY

• US 20120136841A1
• Filed: 11/27/2010
• Published: 05/31/2012
• Est. Priority Date: 11/27/2010
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Balaraman, Giribabu, AMBAT, GOPAKUMAR
• Granted Patent: US 8,504,531 B2
• Time in Patent Office: Days
• US Class Current: 707/692
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays

Abstract

A system and method for application aware de-duplication (de-dup) of data blocks in a virtualized storage array is disclosed. In one embodiment, in a method of application aware de-dup of data blocks on virtualized storage arrays in a storage area network, a de-dup agent is enabled on each of one or more components of the storage area network. A master list of metadata associated with indexed data is then created and stored in the virtualized storage arrays. One or more sublists of metadata are then created from the masterlist and are stored. Upon receiving a write request from an application residing in the host device, it is determined whether data block being written has an entry in a sublist stored in a host device, and if so, the data block is then replaced with a pointer indicating where the data block is residing in the virtualized storage arrays.

Claims

1. A method for application aware de-duplication (de-dup) of data blocks on virtualized storage arrays in a storage area network (SAN), comprising:

enabling a de-dup agent on each of one or more components of the SAN, wherein the one or more components of the SAN comprises a host device, a data path module (DPM), and virtualized storage arrays;

creating a master list of metadata associated with indexed data and storing the masterlist in the virtualized storage arrays;

creating one or more sublists of metadata from the masterlist and storing the one or more sublists in remaining one or more components of the SAN;

upon receiving a write request from an application residing in the host device, determining whether a data block being written has an entry in the sublist stored in the host device; and

if so, replacing the data block with a pointer indicating where the data block is residing in the virtualized storage arrays.

2. The method of claim 1, further comprising:

if not, determining whether the data block being written has an entry in the sublist stored in the DPM; and

if it is determined that the data block being written has an entry in the sublist stored in the DPM, replacing the data block with a pointer indicating where the data block is residing in the virtualized storage arrays.

3. The method of claim 2, further comprising:

if it is determined that the data block being written has no entry in the sublist stored in the DPM, determining whether the data block being written is in the masterlist stored in the virtualized storage arrays;

if it is determined that the data block being written is in the masterlist stored in the virtualized storage arrays, replacing the data block with a pointer indicating where the data block is residing in the virtualized storage arrays; and

if it is determined that the data block being written is not in the masterlist stored in the virtualized storage arrays, writing the data block in one of the virtualized storage arrays.

4. The method of claim 3, further comprising updating the masterlist in the virtualized storage arrays to include metadata associated with the written data block.

5. The method of claim 1, wherein the master list of metadata is an ordered weightage list decided based on number of occurrences of data blocks in each of the virtualized storage arrays.

6. The method of claim 1, wherein the virtualized storage arrays comprise thin provisioned virtual volumes.

7. A SAN, comprising:

a host device;

a DPM connected to the host device; and

one or more virtualized storage arrays connected to the DPM, wherein each of the host device, the DPM and the one or more virtualized storage arrays includes an associated de-dup agent to enable application aware de-dup of data blocks on the one or more virtualized storage arrays in the SAN.

8. The SAN of claim 7, wherein the de-dup agent associated with the one or more virtualized storage arrays creates a master list of metadata associated with indexed data and stores the masterlist in the one or more virtualized storage arrays.

9. The SAN of claim 8, wherein the de-dup agent associated with the host device and the DPM obtains sublists of metadata from the masterlist and stores the sublists in host device and the DPM.

10. The SAN of claim 9, wherein the de-dup agent associated with the host device determines whether a data block being written has an entry in the sublist stored in the host device upon receiving a write request from an application residing in the host device, and wherein the de-dup agent replaces the data block with a pointer indicating where the data block is residing in the virtualized storage arrays, if it is determined that the data block being written has an entry in the sublist stored in the host device.

11. The SAN of claim 10, wherein the de-dup agent associated with the DPM determines whether the data block being written has an entry in the sublist stored in the DPM, if it is determined that the data block being written has no entry in the sublist stored in the host device, and wherein the de-dup agent replaces the data block with a pointer indicating where the data block is residing in the virtualized storage arrays, if it is determined that the data block being written has an entry in the sublist stored in the DPM.

12. The SAN of claim 11, wherein the de-dup agent associated with the virtualized storage arrays determines whether the data block being written is in the masterlist stored in the virtualized storage arrays, if it is determined that the data block being written has no entry in the sublist stored in the DPM, wherein the de-dup agent replaces the data block with a pointer indicating where the data block is residing in the virtualized storage arrays, if it is determined that the data block being written is in the masterlist stored in the virtualized storage arrays, and wherein the de-dup agent writes the data block in one of the virtualized storage arrays, if it is determined that the data block being written is not in the masterlist stored in the virtualized storage arrays.

13. A SAN, comprising:

a host device; and

one or more virtualized storage arrays connected to the host device, wherein each of the host device, and the one or more virtualized storage arrays includes an associated de-dup agent to enable application aware de-dup of data blocks on the one or more virtualized storage arrays in the SAN.

14. The SAN of claim 13, wherein the de-dup agent associated with the one or more virtualized storage arrays creates a master list of metadata associated with indexed data and stores the masterlist in the one or more virtualized storage arrays.

15. The SAN of claim 14, wherein the de-dup agent associated with the host device obtains sublists of metadata from the masterlist and stores the sublists in host device.

16. The SAN of claim 15, wherein the de-dup agent associated with the host device determines whether a data block being written has an entry in the sublist stored in the host device upon receiving a write request from an application residing in the host device, and wherein the de-dup agent replaces the data block with a pointer indicating where the data block is residing in the virtualized storage arrays, if it is determined that the data block being written has an entry in the sublist stored in the host device.

17. The SAN of claim 16, wherein the de-dup agent associated with the virtualized storage arrays determines whether the data block being written is in the masterlist stored in the virtualized storage arrays, if it is determined that the data block being written has no entry in the sublist stored in the host device, wherein the de-dup agent replaces the data block with a pointer indicating where the data block is residing in the virtualized storage arrays, if it is determined that the data block being written is in the masterlist stored in the virtualized storage arrays, and wherein the de-dup agent writes the data block in one of the virtualized storage arrays, if it is determined that the data block being written is not in the masterlist stored in the virtualized storage arrays.

18. A non-transitory computer-readable storage medium for application aware de-dup of data blocks on virtualized storage arrays in a SAN, having instructions that, when executed by a computing device causes the computing device to:

enable a de-dup agent on each of one or more components of the SAN, wherein the one or more components of the storage array comprises a host device, a DPM, and virtualized storage arrays;

create a master list of metadata associated with indexed data and storing the masterlist in the virtualized storage arrays;

create one or more sublists of metadata from the masterlist and storing the one or more sublists in remaining one or more components of the SAN;

upon receiving a write request from an application residing in the host device, determines whether data block being written has an entry in a sublist stored in the host device; and

if so, replaces the data block with a pointer indicating where the data block is residing in the virtualized storage arrays.

19. The non-transitory computer-readable storage medium 18, wherein the master list of metadata is an ordered weightage list decided based on number of occurrences of the data blocks in each of the virtualized storage arrays.

20. The non-transitory computer-readable storage medium 18, wherein the virtualized storage arrays comprise thin provisioned virtual volumes.