Dynamic deduplication

• US 8,688,651 B2
• Filed: 01/25/2011
• Issued: 04/01/2014
• Est. Priority Date: 01/25/2011
• Status: Active Grant

• Current Assignee: Hitachi Vantara, LLC (Hitachi, Ltd.)
• Original Assignee: Sepaton Incorporated (Hitachi, Ltd.)
• Inventors: Kennedy, Jon Christopher, Trimble, Ronald Ray
• Primary Examiner(s): LE, HUNG D
• Application Number:
  US13/013,409
• Publication Number:
  US 20120191670A1
• Time in Patent Office: 1,162 Days
• Field of Search: None
• US Class Current: 707/692
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 2201/81 Threshold

Abstract

Described are computer-based methods and apparatuses, including computer program products, for dynamic deduplication. Data is processed using an algorithm that deduplicates the data based on a first set of parameters. A first moving average of a first deduplication performance metric is calculated for the algorithm over a time period. A second moving average of a second deduplication performance metric is calculated for the algorithm over the time period. It is determined that the first moving average satisfies a first criterion, the second moving average satisfies a second criterion, or both. The algorithm is reconfigured based on a second set of parameters to deduplicate data.

Claims

1. A computerized method for storing data comprising:

processing, by a computing device, data using a deduplication process that deduplicates the data based on a first set of parameters;

calculating, by the computing device, a first moving average of a first deduplication performance metric for the deduplication process over a time period;

calculating, by the computing device, a second moving average of a second deduplication performance metric for the deduplication process over the time period, the second deduplication performance metric being different from the first deduplication performance metric;

determining, by the computing device, the first moving average satisfies a first criterion, the second moving average satisfies a second criterion, or both; and

reconfiguring, by the computing device, the deduplication process based on a second set of parameters to deduplicate data.

2. The method of claim 1 wherein:

calculating the first moving average comprises calculating a first exponentially weighted moving average of the first performance metric over the time period; and

calculating the second moving average comprises calculating a second exponentially weighted moving average of the second performance metric over the time period.

3. The method of claim 2 wherein calculating the first exponentially weighted moving average comprises calculating the first exponentially weighted moving average based on:

a first performance metric value at a current time;

an exponentially weighted moving average for the first performance metric at a previous time; and

an update parameter.

4. The method of claim 2 wherein calculating the second exponentially weighted moving average comprises calculating the second exponentially weighted moving average based on:

a second performance metric value at a current time;

an exponentially weighted moving average for the second performance metric at a previous time; and

an update parameter.

5. The method of claim 1 wherein the first set of parameters causes the deduplication process to deduplicate the data based on a first operating mode, and wherein the second set of parameters causes the deduplication process to deduplicate the data based on a second operating mode.

6. The method of claim 1 wherein reconfiguring comprises generating the second set of parameters by adjusting one or more parameters in the first set of parameters.

7. The method of claim 6 wherein adjusting comprises changing a fingerprint match parameter in the first set of parameters from a first value to a second value.

8. The method of claim 1 comprising:

calculating a third moving average of the first deduplication performance metric over a second time period;

calculating a fourth moving average of the second deduplication performance metric over the second time period;

determining the third moving average satisfies the first criterion, the fourth moving average satisfies the second criterion, or both; and

reconfiguring the deduplication process based on the first set of parameters to deduplicate data.

9. The method of claim 1 wherein the first performance metric is a deduplication speed metric for the deduplication process and the second performance metric is a deduplication ratio metric for the deduplication process.

10. A computer program product, tangibly embodied in a non-transitory computer readable medium, the computer program product including instructions being configured to cause a data processing apparatus to:

process data using a deduplication process that deduplicates the data based on a first set of parameters;

calculate a first moving average of a first deduplication performance metric for the deduplication process over a time period;

calculate a second moving average of a second deduplication performance metric for the deduplication process over the time period, the second deduplication performance metric being different from the first deduplication performance metric;

determine the first moving average satisfies a first criterion, the second moving average satisfies a second criterion, or both; and

reconfigure the deduplication process based on a second set of parameters to deduplicate data.

11. The computer program product of claim 10, wherein the instructions are further configured to cause the data processing apparatus to:

calculate the first moving average as a first exponentially weighted moving average of the first performance metric over the time period; and

calculate the second moving average as a second exponentially weighted moving average of the second performance metric over the time period.

12. An apparatus comprising a processor and memory configured to:

process data using a deduplication process that deduplicates the data based on a first set of parameters;

calculate a first moving average of a first deduplication performance metric for the deduplication process over a time period;

calculate a second moving average of a second deduplication performance metric for the deduplication process over the time period, the second deduplication performance metric being different from the first deduplication performance metric;

determine the first moving average satisfies a first criterion, the second moving average satisfies a second criterion, or both; and

reconfigure the deduplication process based on a second set of parameters to deduplicate data.

13. The apparatus of claim 12, wherein the apparatus is further configured to:

calculate the first moving average as a first exponentially weighted moving average of the first performance metric over the time period; and

calculate the second moving average as a second exponentially weighted moving average of the second performance metric over the time period.

14. The apparatus of claim 13, wherein the apparatus is further configured to calculate the first exponentially weighted moving average based on:

a first performance metric value at a current time;

an exponentially weighted moving average for the first performance metric at a previous time; and

an update parameter.

15. The apparatus of claim 13, wherein the apparatus is further configured to calculate the second exponentially weighted moving average based on:

a second performance metric value at a current time;

an exponentially weighted moving average for the second performance metric at a previous time; and

an update parameter.

16. The apparatus of claim 12, wherein the first set of parameters causes the apparatus to deduplicate the data based on a first operating mode, and wherein the second set of parameters causes the apparatus to deduplicate the data based on a second operating mode.

17. The apparatus of claim 12 wherein the apparatus is further configured to reconfigure the deduplication process by adjusting one or more parameters in the first set of parameters to generate the second set of parameters.

18. The apparatus of claim 17 wherein the apparatus is further configured to adjust the one or more parameters by changing a fingerprint match parameter in the first set of parameters from a first value to a second value.

19. The apparatus of claim 12 wherein the apparatus is further configured to:

calculate a third moving average of the first deduplication performance metric over a second time period;

calculate a fourth moving average of the second deduplication performance metric over the second time period;

determine the third moving average satisfies the first criterion, the fourth moving average satisfies the second criterion, or both; and

reconfigure the deduplication process based on the first set of parameters to deduplicate data.

20. The apparatus of claim 12 wherein the first performance metric is a deduplication speed metric for the deduplication process and the second performance metric is a deduplication ratio metric for the deduplication process.