Policy based tiered data deduplication strategy

• US 7,567,188 B1
• Filed: 04/10/2008
• Issued: 07/28/2009
• Est. Priority Date: 04/10/2008
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: International Business Machines Corporation
• Inventors: Anglin, Matthew J., Cannon, David M., Dawson, Colin S., Martin, Howard N.
• Primary Examiner(s): Mai; Lam T
• Application Number:
  US12/100,695
• Time in Patent Office: 474 Days
• Field of Search: 341 60- 90, 707/3, 707/6, 707/7, 707/101, 707/202, 707/204, 707/104.1
• US Class Current: 341/63
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/1756 based on delta files
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: H03M 7/3084 using adaptive string matching, e.g. the Lempel-Ziv method
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

The present invention provides for a method, system, and computer program for the application of data deduplication according to a policy-based strategy of tiered data. The method operates by defining a plurality of data storage policies for data in a deduplication system, policies which may be arranged in tiers. Data objects are classified according to a selected data storage policy and are split into data chunks. If the selected data storage policy for the data object does not allow deduplication, the data chunks are stored in a deduplication pool. If the selected data storage policy for the data object allows deduplication, deduplication is performed. The data storage policy may specify a maximum number of references to data chunks, facilitating storage of new copies of the data chunks when the maximum number of references is met.

Claims

1. A method for applying a deduplication strategy to a data object based on a data storage policy, comprising:

defining a plurality of data storage policies for a deduplication pool, each data storage policy containing settings including a maximum reference count for data chunks;

classifying the data object within a selected data storage policy;

dividing the data object into a plurality of data chunks, each data chunk having reference count data to track a number of references thereto;

storing each data chunk of the data object in the deduplication pool if the selected data storage policy does not allow deduplication of the data object; and

performing deduplication on the data object if the selected data storage policy allows deduplication of the data object, including for each data chunk of the data object;

initializing the data chunk reference count data and storing the data chunk in the deduplication pool if a previously stored identical copy of the data chunk does not exist in the deduplication pool,

updating the reference count data of and creating a pointer to a previously stored identical copy of the data chunk if the previously stored identical copy of the data chunk exists in the deduplication pool and has a reference count less than the selected data storage policy maximum reference count, and

initializing the data chunk reference count data and storing the data chunk in the deduplication pool if each previously stored identical copy of the data chunk existing within the deduplication pool contains a reference count equal to or greater than the selected data storage policy maximum reference count.

2. The method of claim 1, wherein the plurality of data storage policies includes at least three tiered data storage policies, a first data storage policy not allowing data deduplication of data objects, a second data storage policy allowing deduplication of data objects and containing a policy limit for references, and a third data storage policy allowing deduplication of data objects and containing no policy limit for references.

3. The method of claim 1, wherein the plurality of data storage policies includes a data storage policy that does not allow deduplication of data objects.

4. The method of claim 1, wherein the plurality of data storage policies includes a data storage policy that allows deduplication of data objects and contains a defined policy limit for references to data chunks.

5. The method of claim 1, wherein the plurality of data storage policies includes a data storage policy that allows deduplication of data objects and contains no policy limit for references to data chunks.

6. The method of claim 1, wherein the plurality of data storage policies is arranged in tiers, and data chunks classified according to a data storage policy are permitted to create references to data chunks within the same data storage policy or a higher tiered data storage policy and are not permitted to create references to data chunks within a lower tiered data storage policy.

7. The method of claim 1, wherein the step of performing deduplication on the data object occurs synchronously during ingestion of the data object into the deduplication pool.

8. The method of claim 1, wherein the step of performing deduplication on the data object occurs asynchronously during an operation subsequent to ingestion of the data object into the deduplication pool.

9. The method of claim 1, wherein the reference count data includes a plurality of reference counts for tracking the number of references to the data chunk from data chunks classified according to each of the plurality of data storage policies.

10. A method for applying a deduplication strategy to a data object based on a data storage policy, comprising:

defining a first data storage policy for a deduplication pool;

classifying a first data object within the first data storage policy;

dividing the first data object into a plurality of data chunks, each data chunk having reference count data to track a number of references by storage policy thereto;

storing each data chunk of the first data object in the deduplication pool in accordance with the first data storage policy;

defining a second data storage policy for the deduplication pool, the second data storage policy including a maximum reference count for data chunks;

classifying a second data object within the second data storage policy;

dividing the second data object into a plurality of data chunks, each data chunk having reference count data to track a number of references by storage policy thereto; and

performing deduplication on the second data object in accordance with the second data storage policy, including for each data chunk of the second data object;

initializing the data chunk reference count data and storing the data chunk in the deduplication pool if a previously stored identical copy of the data chunk does not exist in the deduplication pool,

updating the reference count data of and creating a reference to a previously stored identical copy of the data chunk if the previously stored identical copy of the data chunk exists in the deduplication pool and has a reference count less than the second data storage policy maximum reference count, and

initializing the data chunk reference count data and storing the data chunk in the deduplication pool if each previously stored identical copy of the data chunk existing within the deduplication pool contains a reference count equal to or greater than the second data storage policy maximum reference count.

11. A method for applying a deduplication strategy to a data object based on a data storage policy, comprising:

performing deduplication on the data object if the data storage policy associated with the data object allows deduplication, including for each data chunk of the data object;

updating reference count data of and creating a reference to a previously stored identical copy of the data chunk if the previously stored identical copy of the data chunk has a reference count less than a maximum reference count of the data storage policy, and

storing the data chunk and initializing the reference count data of the data chunk if each previously stored identical copy of the data chunk contains a reference count equal to or greater than the maximum reference count of the data storage policy.

12. A computer program product comprising a computer useable medium having a computer readable program for applying a deduplication strategy to a data object based on a data storage policy, wherein the computer readable program when executed on a computer causes the computer to:

define a plurality of data storage policies for a deduplication pool, each data storage policy containing settings including a maximum reference count for data chunks;

classify the data object within a selected data storage policy;

divide the data object into a plurality of data chunks, each data chunk having reference count data to track a number of references thereto;

store each data chunk of the data object in the deduplication pool if the selected data storage policy does not allow deduplication of the data object; and

perform deduplication on the data object if the selected data storage policy allows deduplication of the data object, including for each data chunk of the data object;

initializing the data chunk reference count data and storing the data chunk in the deduplication pool if a previously stored identical copy of the data chunk does not exist in the deduplication pool,

updating the reference count data of and creating a pointer to a previously stored identical copy of the data chunk if the previously stored identical copy of the data chunk exists in the deduplication pool and has a reference count less than the selected data storage policy maximum reference count, and

initializing the data chunk reference count data and storing the data chunk in the deduplication pool if each previously stored identical copy of the data chunk existing within the deduplication pool contains a reference count equal to or greater than the selected data storage policy maximum reference count.

13. The computer program product as in claim 12, wherein the plurality of data storage policies includes at least three tiered data storage policies, a first data storage policy not allowing data deduplication of data objects, a second data storage policy allowing deduplication of data objects and containing a policy limit for references, and a third data storage policy allowing deduplication of data objects and containing no policy limit for references.

14. The computer program product as in claim 12, wherein the plurality of data storage policies includes a data storage policy that does not allow deduplication of data objects.

15. The computer program product as in claim 12, wherein the plurality of data storage policies includes a data storage policy that allows deduplication of data objects and contains a defined policy limit for references to data chunks.

16. The computer program product as in claim 12, wherein the plurality of data storage policies includes a data storage policy that allows deduplication of data objects and contains no policy limit for references to data chunks.

17. The computer program product as in claim 12, wherein the plurality of data storage policies is arranged in tiers, and data chunks classified according to a data storage policy are permitted to create references to data chunks within the same data storage policy or a higher tiered data storage policy and are not permitted to create references to data chunks within a lower tiered data storage policy.

18. The computer program product as in claim 12, wherein the reference count data includes a plurality of reference counts for tracking the number of references to the data chunk from data chunks classified according to each of the plurality of data storage policies.

19. A system, comprising:

at least one processor; and

at least one memory storing instructions operable with the at least one processor for applying a deduplication strategy to a data object based on a data storage policy, the instructions being executed for;

defining a plurality of data storage policies for a deduplication pool, each data storage policy containing settings including a maximum reference count for data chunks;

classifying the data object within a selected data storage policy;

dividing the data object into a plurality of data chunks, each data chunk having a reference count data to track a number of references thereto;

storing each data chunk of the data object in the deduplication pool if the selected data storage policy does not allow deduplication of the data object; and

performing deduplication on the data object if the selected data storage policy allows deduplication of the data object, including for each data chunk of the data object;

initializing the data chunk reference count data and storing the data chunk in the deduplication pool if a previously stored identical copy of the data chunk does not exist in the deduplication pool,

updating the reference count data of and creating a pointer to a previously stored identical copy of the data chunk if the previously stored identical copy of the data chunk exists in the deduplication pool and has a reference count less than the selected data storage policy maximum reference count, and

initializing the data chunk reference count data and storing the data chunk in the deduplication pool if each previously stored identical copy of the data chunk existing within the deduplication pool contains a reference count equal to or greater than the selected data storage policy maximum reference count.

20. The system as in claim 19, wherein the plurality of data storage policies includes at least three tiered data storage policies, a first data storage policy not allowing data deduplication of data objects, a second data storage policy allowing deduplication of data objects and containing a policy limit for references, and a third data storage policy allowing deduplication of data objects and containing no policy limit for references.

21. The system as in claim 19, wherein the plurality of data storage policies includes a data storage policy that does not allow deduplication of data objects.

22. The system as in claim 19, wherein the plurality of data storage policies includes a data storage policy that allows deduplication of data objects and contains a defined policy limit for references to data chunks.

23. The system as in claim 19, wherein the plurality of data storage policies includes a data storage policy that allows deduplication of data objects and contains no policy limit for references to data chunks.

24. The system as in claim 19, wherein the plurality of data storage policies is arranged in tiers, and data chunks classified according to a data storage policy are permitted to create references to data chunks within the same data storage policy or a higher tiered data storage policy and are not permitted to create references to data chunks within a lower tiered data storage policy.

25. The system as in claim 19, wherein the reference count data includes a plurality of reference counts for tracking the number of references to the data chunk from data chunks classified according to each of the plurality of data storage policies.