DEDUPLICATED DATA PROCESSING RATE CONTROL

• US 20110040951A1
• Filed: 08/11/2009
• Published: 02/17/2011
• Est. Priority Date: 08/11/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: KLIPPER, Lior, SONIN, Oded, AKIRAV, Shay H., BACHAR, Yariv, ASHER, Ron
• Granted Patent: US 8,385,192 B2
• Time in Patent Office: Days
• US Class Current: 712/4
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 16/1756 based on delta files
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: H04L 47/10 Flow control; Congestion control

Abstract

Various embodiments for deduplicated data processing rate control using at least one processor device in a computing environment are provided. A plurality of workers is configured for parallel processing of deduplicated data entities in a plurality of chunks. The deduplicated data processing rate is regulated using a rate control mechanism. The rate control mechanism incorporates a debt/credit algorithm specifying which of the plurality of workers processing the deduplicated data entities must wait for each of a plurality of calculated required sleep times. The rate control mechanism is adapted to limit a data flow rate based on a penalty acquired during a last processing of one of the plurality of chunks in a retroactive manner, and further adapted to operate on at least one vector representation of at least one limit specification to accommodate a variety of available dimensions corresponding to the at least one limit specification.

Claims

1. A method for deduplicated data processing rate control using at least one processor device in a computing environment, the method comprising:

configuring a plurality of workers for parallel processing of deduplicated data entities in a plurality of chunks; and

regulating a deduplicated data processing rate using a rate control mechanism, the rate control mechanism incorporating a debt/credit algorithm specifying which of the plurality of workers processing the deduplicated data entities must wait for each of a plurality of calculated required sleep times

2. The method of claim 1, wherein the rate control mechanism is adapted to limit a data flow rate based on a penalty acquired during a last processing of one of the plurality of chunks in a retroactive manner, and further adapted to operate on at least one vector representation of at least one limit specification to accommodate a variety of available dimensions corresponding to the at least one limit specification.

3. The method of claim 1, further including defining the at least one vector representation to represent the variety of available dimensions corresponding to the at least one limit specification.

4. The method of claim 3, further including, pursuant to defining the at least one vector representation, defining a plurality of vector operations for the at least one vector representation including addition, subtraction, equality, and assignment vector operations, wherein the subtraction vector operation is defined to reset to a zero value when subtracted below zero.

5. The method of claim 4, further including, pursuant to defining the at least one vector representation, defining a zero vector for evaluating a non-zero delta vector of the at least one vector representation.

6. The method of claim 5, further including, pursuant to defining the at least one vector representation, using the at least one vector representation to define a limits per time unit vector, wherein the variety of available dimensions are commonly measured in a per time unit corresponding to the limits per time unit vector.

7. The method of claim 6, further including initializing the rate control mechanism by performing at least one of:

setting a maximum allowable sleep time to one of a predefined finite value and a predefined infinite value, and

setting initial values for a last update time, a current debt vector, a limits per time vector, and a maximum debt vector.

8. The method of claim 7, further including, pursuant to regulating the deduplicated data processing rate, performing at least one of:

retrieving a current system time value,

generating a sample vector as the at least one vector representation for the current system time value,

using the current system time value, calculating an elapsed time credit since the last update time, and

updating the last update time.

9. The method of claim 8, further including, pursuant to regulating the deduplicated data processing rate, performing at least one of:

adding the sample vector to the current debt vector and subtracting the elapsed time credit to obtain an updated debt vector,

subtracting the maximum debt vector from the current debt vector to obtain a delta vector.

10. The method of claim 9, further including, pursuant to regulating the deduplicated data processing rate, if the delta vector is non-zero,

calculating a required sleep time as a best fit between the maximum allowable sleep time, the delta vector, and the limit per time vector, wherein the required sleep time is one of the plurality of calculated required sleep times.

11. The method of claim 10, further including refraining from processing the deduplicated data entities pursuant to the required sleep time.

12. A system for deduplicated data processing rate control in a computing environment, the method comprising:

at least one processor device operable in the computing environment, wherein the at least one processor device is adapted for;

configuring a plurality of workers for parallel processing of deduplicated data entities in a plurality of chunks, and

regulating a deduplicated data processing rate using a rate control mechanism, the rate control mechanism incorporating a debt/credit algorithm specifying which of the plurality of workers processing the deduplicated data entities must wait for each of a plurality of calculated required sleep times;

wherein;

the rate control mechanism is adapted to limit a data flow rate based on a penalty acquired during a last processing of one of the plurality of chunks in a retroactive manner, and

the rate control mechanism is further adapted to operate on at least one vector representation of at least one limit specification to accommodate a variety of available dimensions corresponding to the at least one limit specification.

13. The system of claim 12, wherein the at least one processor device is further adapted for defining the at least one vector representation to represent the variety of available dimensions corresponding to the at least one limit specification.

14. The system of claim 13, wherein the at least one processor device is further adapted for, pursuant to defining the at least one vector representation, defining a plurality of vector operations for the at least one vector representation including addition, subtraction, equality, and assignment vector operations, wherein the subtraction vector operation is defined to reset to a zero value when subtracted below zero.

15. The system of claim 14, wherein the at least one processor device is further adapted for, pursuant to defining the at least one vector representation, defining a zero vector for evaluating a non-zero delta vector of the at least one vector representation.

16. The system of claim 15, wherein the at least one processor device is further adapted for, pursuant to defining the at least one vector representation, using the at least one vector representation to define a limits per time unit vector, wherein the variety of available dimensions are commonly measured in a per time unit corresponding to the limits per time unit vector.

17. The system of claim 16, wherein the at least one processor device is further adapted for initializing the rate control mechanism by performing at least one of:

setting a maximum allowable sleep time to one of a predefined finite value and a predefined infinite value, and

setting initial values for a last update time, a current debt vector, a limits per time vector, and a maximum debt vector.

18. The system of claim 17, wherein the at least one processor device is further adapted for, pursuant to regulating the deduplicated data processing rate, performing at least one of:

retrieving a current system time value,

generating a sample vector as the at least one vector representation for the current system time value,

using the current system time value, calculating an elapsed time credit since the last update time, and

updating the last update time.

19. The system of claim 18, wherein the at least one processor device is further adapted for, pursuant to regulating the deduplicated data processing rate, performing at least one of:

adding the sample vector to the current debt vector and subtracting the elapsed time credit to obtain an updated debt vector,

subtracting the maximum debt vector from the current debt vector to obtain a delta vector.

20. The system of claim 19, wherein the at least one processor device is further adapted for, pursuant to regulating the deduplicated data processing rate, if the delta vector is non-zero,

calculating a required sleep time as a best fit between the maximum allowable sleep time, the delta vector, and the limit per time vector, wherein the required sleep time is one of the plurality of calculated required sleep times.

21. The system of claim 20, wherein the at least one processor device is further adapted for refraining from processing the deduplicated data entities pursuant to the required sleep time.

22. The system of claim 12, wherein a first of the plurality of workers is configured to be locally operable to process one of the deduplicated data entities, and a second of the plurality of workers is configured to be remotely operable to process the one of the deduplicated data entities.

23. The system of claim 12, wherein first and second workers of the plurality of workers are configured to be operable to process one of the deduplicated data entities.

24. A computer program product for deduplicated data processing rate control using at least one processor device in a computing environment, the computer program product comprising a computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

a first executable portion for configuring a plurality of workers for parallel processing of deduplicated data entities in a plurality of chunks; and

a second executable portion for regulating a deduplicated data processing rate using a rate control mechanism, the rate control mechanism incorporating a debt/credit algorithm specifying which of the plurality of workers processing the deduplicated data entities must wait for each of a plurality of calculated required sleep times;

wherein;

the rate control mechanism is adapted to limit a data flow rate based on a penalty acquired during a last processing of one of the plurality of chunks in a retroactive manner, and

the rate control mechanism is further adapted to operate on at least one vector representation of at least one limit specification to accommodate a variety of available dimensions corresponding to the at least one limit specification.

25. The computer program product of claim 24, further including a third executable portion for defining the at least one vector representation to represent the variety of available dimensions corresponding to the at least one limit specification.

26. The computer program product of claim 25, further including a fourth executable portion for, pursuant to defining the at least one vector representation, defining a plurality of vector operations for the at least one vector representation including addition, subtraction, equality, and assignment vector operations, wherein the subtraction vector operation is defined to reset to a zero value when subtracted below zero.

27. The computer program product of claim 26, further including a fifth executable portion for, pursuant to defining the at least one vector representation, defining a zero vector for evaluating a non-zero delta vector of the at least one vector representation.

28. The computer program product of claim 27, further including a sixth executable portion for, pursuant to defining the at least one vector representation, using the at least one vector representation to define a limits per time unit vector, wherein the variety of available dimensions are commonly measured in a per time unit corresponding to the limits per time unit vector.

29. The computer program product of claim 26, further including a seventh executable portion for initializing the rate control mechanism by performing at least one of:

setting a maximum allowable sleep time to one of a predefined finite value and a predefined infinite value, and

setting initial values for a last update time, a current debt vector, a limits per time vector, and a maximum debt vector.

30. The computer program product of claim 29, further including an eighth executable portion for, pursuant to regulating the deduplicated data processing rate, performing at least one of:

retrieving a current system time value,

generating a sample vector as the at least one vector representation for the current system time value,

using the current system time value, calculating an elapsed time credit since the last update time, and

updating the last update time.

31. The computer program product of claim 30, further including a ninth executable portion for, pursuant to regulating the deduplicated data processing rate, performing at least one of:

adding the sample vector to the current debt vector and subtracting the elapsed time credit to obtain an updated debt vector,

subtracting the maximum debt vector from the current debt vector to obtain a delta vector.

32. The computer program product of claim 31, further including a tenth executable portion for, pursuant to regulating the deduplicated data processing rate, if the delta vector is non-zero,

calculating a required sleep time as a best fit between the maximum allowable sleep time, the delta vector, and the limit per time vector, wherein the required sleep time is one of the plurality of calculated required sleep times.

33. The computer program product of claim 32, further including an eleventh executable portion for refraining from processing the deduplicated data entities pursuant to the required sleep time.