Deduplicated data processing rate control

• US 9,086,814 B2
• Filed: 02/06/2013
• Issued: 07/21/2015
• Est. Priority Date: 08/11/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Akirav, Shay H., Asher, Ron, Bachar, Yariv, Klipper, Lior, Sonin, Oded
• Primary Examiner(s): Petranek, Jacob A
• Application Number:
  US13/760,963
• Publication Number:
  US 20130144848A1
• Time in Patent Office: 895 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 16/1756 based on delta files
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: H04L 47/10 Flow control; Congestion control

Abstract

Workers are configured for parallel processing of deduplicated data entities in chunks. The deduplicated data processing rate is regulated using a rate control mechanism. The rate control mechanism incorporates a debt/credit algorithm specifying which of the workers processing the deduplicated data entities must wait for each of a multiplicity of calculated required sleep times.

Claims

1. A system for deduplicated data processing rate control in a computing environment, the system comprising:

a rate control mechanism;

at least one processor device, in communication with the rate control mechanism, operable in the computing environment for controlling the rate control mechanism, wherein the at least one processor device;

configures a plurality of workers for parallel processing of deduplicated data entities in a plurality of chunks, and

regulates a deduplicated data processing rate using a rate control mechanism, the rate control mechanism incorporating a debt/credit algorithm specifying which of the plurality of workers processing the deduplicated data entities must wait for each of a plurality of calculated required sleep times.

2. The system of claim 1, wherein the rate control mechanism:

limits a data flow rate based on a penalty acquired during a last processing of one of the plurality of chunks in a retroactive manner, and

operates on at least one vector representation of at least one limit specification to accommodate a variety of available dimensions corresponding to the at least one limit specification.

3. The system of claim 1, wherein the rate control mechanism defines the at least one vector representation to represent the variety of available dimensions corresponding to the at least one limit specification.

4. The system of claim 3, wherein the rate control mechanism, pursuant to defining the at least one vector representation, defines a plurality of vector operations for the at least one vector representation including addition, subtraction, equality, and assignment vector operations, wherein the subtraction vector operation is defined to reset to a zero value when subtracted below zero.

5. The system of claim 4, wherein the rate control mechanism, pursuant to defining the at least one vector representation, defines a zero vector for evaluating a non-zero delta vector of the at least one vector representation.

6. The system of claim 5, wherein the rate control mechanism, pursuant to defining the at least one vector representation, uses the at least one vector representation to define a limits per time unit vector, wherein the variety of available dimensions are commonly measured in a per time unit corresponding to the limits per time unit vector.

7. The system of claim 6, wherein the at least one processor device initiates the rate control mechanism by performing at least one of:

setting a maximum allowable sleep time to one of a predefined finite value and a predefined infinite value, and

setting initial values for a last update time, a current debt vector, a limits per time vector, and a maximum debt vector.

8. The system of claim 7, wherein the rate control mechanism, pursuant to regulating the deduplicated data processing rate, performs at least one of:

retrieving a current system time value,

generating a sample vector as the at least one vector representation for the current system time value,

using the current system time value, calculating an elapsed time credit since the last update time, and

updating the last update time.

9. The system of claim 8, wherein the rate control mechanism, pursuant to regulating the deduplicated data processing rate, performs at least one of:

adding the sample vector to the current debt vector and subtracting the elapsed time credit to obtain an updated debt vector,

subtracting the maximum debt vector from the current debt vector to obtain a delta vector.

10. The system of claim 9, wherein the rate control mechanism, pursuant to regulating the deduplicated data processing rate, if the delta vector is non-zero,

calculates a required sleep time as a best fit between the maximum allowable sleep time, the delta vector, and the limit per time vector, wherein the required sleep time is one of the plurality of calculated required sleep times.

11. The system of claim 10, wherein the rate control mechanism refrains from processing the deduplicated data entities pursuant to the required sleep time.

12. A computer program product for deduplicated data processing rate control in a computing environment by a processor device, the computer program product comprising a non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

a first executable portion that configures a plurality of workers for parallel processing of deduplicated data entities in a plurality of chunks; and

a second executable portion that regulates a deduplicated data processing rate using a rate control mechanism, the rate control mechanism incorporating a debt/credit algorithm specifying which of the plurality of workers processing the deduplicated data entities must wait for each of a plurality of calculated required sleep times.

13. The computer program product of claim 12, further including a third executable portion that:

limits a data flow rate based on a penalty acquired during a last processing of one of the plurality of chunks in a retroactive manner, and

operates on at least one vector representation of at least one limit specification to accommodate a variety of available dimensions corresponding to the at least one limit specification.

14. The computer program product of claim 12, further including a third executable portion that defines the at least one vector representation to represent the variety of available dimensions corresponding to the at least one limit specification.

15. The computer program product of claim 14, further including a fourth executable portion that, pursuant to defining the at least one vector representation, defines a plurality of vector operations for the at least one vector representation including addition, subtraction, equality, and assignment vector operations, wherein the subtraction vector operation is defined to reset to a zero value when subtracted below zero.

16. The computer program product of claim 15, further including a fifth executable portion that, pursuant to defining the at least one vector representation, defines a zero vector for evaluating a non-zero delta vector of the at least one vector representation.

17. The computer program product of claim 16, further including a sixth executable portion that, pursuant to defining the at least one vector representation, uses the at least one vector representation to define a limits per time unit vector, wherein the variety of available dimensions are commonly measured in a per time unit corresponding to the limits per time unit vector.

18. The computer program product of claim 17, further including a seventh executable portion that initiates the rate control mechanism by performing at least one of:

setting a maximum allowable sleep time to one of a predefined finite value and a predefined infinite value, and

setting initial values for a last update time, a current debt vector, a limits per time vector, and a maximum debt vector.

19. The computer program product of claim 18, further including an eighth executable portion that, pursuant to regulating the deduplicated data processing rate, performs at least one of:

retrieving a current system time value,

generating a sample vector as the at least one vector representation for the current system time value,

using the current system time value, calculating an elapsed time credit since the last update time, and

updating the last update time.

20. The computer program product of claim 19, further including a tenth executable portion that, pursuant to regulating the deduplicated data processing rate, performs at least one of:

adding the sample vector to the current debt vector and subtracting the elapsed time credit to obtain an updated debt vector,

subtracting the maximum debt vector from the current debt vector to obtain a delta vector.

21. The computer program product of claim 20, further including an eleventh executable portion that, pursuant to regulating the deduplicated data processing rate, if the delta vector is non-zero,

calculates a required sleep time as a best fit between the maximum allowable sleep time, the delta vector, and the limit per time vector, wherein the required sleep time is one of the plurality of calculated required sleep times.

22. The computer program product of claim 12, further including a third executable portion that refrains from processing the deduplicated data entities pursuant to the required sleep time.