Dynamic and optimized management of grid system resources

• US 10,073,855 B2
• Filed: 05/21/2015
• Issued: 09/11/2018
• Est. Priority Date: 05/21/2015
• Status: Active Grant

• Current Assignee: ExaGrid Systems, Inc.
• Original Assignee: ExaGrid Systems, Inc.
• Inventors: Chao, Yee-ching, Watkins, Peter, Hansen, Thomas G., Smith, Stephen A., Arruda, Luis, VanderSpek, Adrian T.
• Primary Examiner(s): Morrison, Jay A
• Application Number:
  US14/718,702
• Publication Number:
  US 20160342618A1
• Time in Patent Office: 1,209 Days
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1466 to make the backup process non-disruptive
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 9/505 considering the load

Abstract

A system, a method, and a computer program product for or managing grid server system resource(s) are disclosed. A processor of a grid server in a plurality of grid servers determines an ingestion rate of data received by the grid server for a backup. The ingestion rate corresponds to a speed of transferring of the received data into a memory of the grid server. The processor determines a deduplication and replication rate of the received data. Deduplication represents a deduplication of the received data by the grid server. Replication represents replication of the deduplicated received data to another server in the plurality of servers. The processor adjusts the deduplication and replication rate based on the ingestion rate.

Claims

1. A computer implemented method, comprising

sensing, by a processor of a grid server in a plurality of grid servers, an ingestion rate of data received by the grid server for a backup, the ingestion rate corresponding to a speed of transferring of the received data into a memory of the grid server, the data is received from at least one data management application communicatively coupled to the plurality of grid servers;

monitoring, by the processor of the grid server, a deduplication and replication rate of the received data, the ingestion rate is being dependent on the deduplication and replication rate,

the deduplication including deduplicating the received data by the grid server based on a determination that at least one zone stamp of a zone within the received data is similar to at least another zone stamp of another zone within the received data, and

the replication including replicating the deduplicated received data to another server in the plurality of servers, the another server being communicatively coupled to the grid server using a communications network; and

adjusting, by the processor of the grid server, the deduplication and replication rate based on the sensed ingestion rate, wherein the adjusting includes;

monitoring at least one of the following;

a backup process, deduplication and replication processes, a restore process, a copy-to-tape process, a cross-replication process, a purge process, a rebalance process as performed by the grid server; and

performing, based on the monitoring, sampling of at least one of the following;

the ingestion rate, and the deduplication and replication rate;

performing ingestion, deduplication and replication of data in parallel; and

minimizing, based on the sampling and parallel performance of ingestion, deduplication and replication of data, a recovery point objective time associated with the received data, the recovery point objective time representing a difference between times when the data is received into the memory of the grid server and when the data is received into a memory of another grid server.

2. The method according to claim 1, wherein the adjusting further includes increasing the deduplication and replication rate when the ingestion rate is less than or equal to a first predetermined threshold.

3. The method according to claim 2, wherein the adjusting further includes unchanging the deduplication and replication rate when the ingestion rate is greater than the first predetermined threshold and less than or equal to a second predetermined threshold.

4. The method according to claim 3, wherein the adjusting further includes reducing the deduplication and replication rate when the ingestion rate is greater than the second predetermined threshold and less than or equal to a third predetermined threshold.

5. The method according to claim 4, wherein the adjusting further include setting the deduplication and replication rate to a predetermined minimum deduplication and replication rate when the ingestion rate is greater than the third predetermined threshold.

6. The method according to claim 1, further comprising

performing, based on the monitoring and the sampling, using the grid server, at least one of the following:

increasing the deduplication and replication rate, decreasing the deduplication and replication rate, increasing the ingestion rate, decreasing the ingestion rate, increasing a protection rate corresponding a combination of the deduplication rate and the replication rate, decreasing the protection rate, and/or any combination thereof.

7. The method according to claim 1, wherein the sampling is performed based on at least one of the following:

periodically, automatically, and during a predetermined time interval.

8. The method according to claim 1, wherein the memories of the grid server and another grid server include at least one of the following:

a non-volatile storage medium and a volatile storage medium.

9. The method according to claim 1, wherein the sampling is performed based on at least one of the following parameter:

amount of data stored in a network socket receive buffer associated with the grid server, data packet round trip time associate with transmission of data between at least two grid servers, transmission control protocol advertised receive buffer size, bandwidth of a network including the grid server, a time to complete at least one disk read/write input/output request associated with the grid server, and/or any combination thereof.

10. The method according to claim 9, further comprising

performing smoothing of the at least one parameter over a predetermined period of time;

generating, based on the smoothing, an ingestion score value;

comparing the generated ingestion score value with a threshold ingestion score value; and

performing, based on the comparing, the adjusting of the deduplication and replication rate.

11. The method according to claim 1, wherein the received data is stored temporarily in a temporary storage location of the grid server.

12. The method according to claim 11, further comprising transferring the received data from the temporary storage location of the grid server for deduplication and replication by the grid server.

13. The method according to claim 11, further comprising performing, based on the received data temporarily stored in the temporary storage location of the grid server, at an increased speed at least one of the following:

generating at least one backup tape, generating at least one virtual machine image, performing an instant recovery of the received data from the temporary storage location, and restoring at least one file.

14. A system comprising:

at least one programmable processor; and

a machine-readable medium storing instructions that, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising;

sensing, by a processor of a grid server in a plurality of grid servers, an ingestion rate of data received by the grid server for a backup, the ingestion rate corresponding to a speed of transferring of the received data into a memory of the grid server, the data is received from at least one data management application communicatively coupled to the plurality of grid servers;

monitoring, by the processor of the grid server, a deduplication and replication rate of the received data, the ingestion rate is being dependent on the deduplication and replication rate,

the deduplication including deduplicating the received data by the grid server based on a determination that at least one zone stamp of a zone within the received data is similar to at least another zone stamp of another zone within the received data, and

the replication including replicating the deduplicated received data to another server in the plurality of servers, the another server being communicatively coupled to the grid server using a communications network; and

adjusting, by the processor of the grid server, the deduplication and replication rate based on the sensed ingestion rate, wherein the adjusting includes;

monitoring at least one of the following;

a backup process, deduplication and replication processes, a restore process, a copy-to-tape process, a cross-replication process, a purge process, a rebalance process as performed by the grid server; and

performing, based on the monitoring, sampling of at least one of the following;

the ingestion rate, and the deduplication and replication rate;

performing ingestion, deduplication and replication of data in parallel; and

minimizing, based on the sampling and parallel performance of ingestion, deduplication and replication of data, a recovery point objective time associated with the received data, the recovery point objective time representing a difference between times when the data is received into the memory of the grid server and when the data is received into a memory of another grid server.

15. The system according to claim 14, wherein the adjusting further includes increasing the deduplication and replication rate when the ingestion rate is less than or equal to a first predetermined threshold.

16. The system according to claim 15, wherein the adjusting further includes unchanging the deduplication and replication rate when the ingestion rate is greater than the first predetermined threshold and less than or equal to a second predetermined threshold.

17. The system according to claim 16, wherein the adjusting further includes reducing the deduplication and replication rate when the ingestion rate is greater than the second predetermined threshold and less than or equal to a third predetermined threshold.

18. The system according to claim 17, wherein the adjusting further include setting the deduplication and replication rate to a predetermined minimum deduplication and replication rate when the ingestion rate is greater than the third predetermined threshold.

19. The system according to claim 14, wherein the operations further comprise

performing, based on the monitoring and the sampling, using the grid server, at least one of the following:

increasing the deduplication and replication rate, decreasing the deduplication and replication rate, increasing the ingestion rate, decreasing the ingestion rate, increasing a protection rate corresponding a combination of the deduplication rate and the replication rate, decreasing the protection rate, and/or any combination thereof.

20. The system according to claim 14, wherein the sampling is performed based on at least one of the following:

periodically, automatically, and during a predetermined time interval.

21. The system according to claim 14, wherein the memories of the grid server and another grid server include at least one of the following:

a non-volatile storage medium and a volatile storage medium.

22. The system according to claim 14, wherein the sampling is performed based on at least one of the following parameter:

amount of data stored in a network socket receive buffer associated with the grid server, data packet round trip time associate with transmission of data between at least two grid servers, transmission control protocol advertised receive buffer size, bandwidth of a network including the grid server, a time to complete at least one disk read/write input/output request associated with the grid server, and/or any combination thereof.

23. The system according to claim 22, wherein the operations further comprise

performing smoothing of the at least one parameter over a predetermined period of time;

generating, based on the smoothing, an ingestion score value;

comparing the generated ingestion score value with a threshold ingestion score value; and

performing, based on the comparing, the adjusting of the deduplication and replication rate.

24. The system according to claim 14, wherein the received data is stored temporarily in a temporary storage location of the grid server.

25. The system according to claim 24, wherein the operations further comprise transferring the received data from the temporary storage location of the grid server for deduplication and replication by the grid server.

26. The system according to claim 24, wherein the operations further comprise performing, based on the received data temporarily stored in the temporary storage location of the grid server, at an increased speed at least one of the following:

generating at least one backup tape, generating at least one virtual machine image, performing an instant recovery of the received data from the temporary storage location, and restoring at least one file.

27. A computer program product comprising a machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:

sensing, by a processor of a grid server in a plurality of grid servers, an ingestion rate of data received by the grid server for a backup, the ingestion rate corresponding to a speed of transferring of the received data into a memory of the grid server, the data is received from at least one data management application communicatively coupled to the plurality of grid servers;

monitoring, by the processor of the grid server, a deduplication and replication rate of the received data, the ingestion rate is being dependent on the deduplication and replication rate,

the deduplication including deduplicating the received data by the grid server based on a determination that at least one zone stamp of a zone within the received data is similar to at least another zone stamp of another zone within the received data, and

the replication including replicating the deduplicated received data to another server in the plurality of servers, the another server being communicatively coupled to the grid server using a communications network; and

adjusting, by the processor of the grid server, the deduplication and replication rate based on the sensed ingestion rate, wherein the adjusting includes;

monitoring at least one of the following;

a backup process, deduplication and replication processes, a restore process, a copy-to-tape process, a cross-replication process, a purge process, a rebalance process as performed by the grid server; and

performing, based on the monitoring, sampling of at least one of the following;

the ingestion rate, and the deduplication and replication rate;

performing ingestion, deduplication and replication of data in parallel; and

minimizing, based on the sampling and parallel performance of ingestion, deduplication and replication of data, a recovery point objective time associated with the received data, the recovery point objective time representing a difference between times when the data is received into the memory of the grid server and when the data is received into a memory of another grid server.

28. The computer program product according to claim 27, wherein the adjusting further includes increasing the deduplication and replication rate when the ingestion rate is less than or equal to a first predetermined threshold.

29. The computer program product according to claim 28, wherein the adjusting further includes unchanging the deduplication and replication rate when the ingestion rate is greater than the first predetermined threshold and less than or equal to a second predetermined threshold.

30. The computer program product according to claim 29, wherein the adjusting further includes reducing the deduplication and replication rate when the ingestion rate is greater than the second predetermined threshold and less than or equal to a third predetermined threshold.

31. The computer program product according to claim 30, wherein the adjusting further include setting the deduplication and replication rate to a predetermined minimum deduplication and replication rate when the ingestion rate is greater than the third predetermined threshold.

32. The computer program product according to claim 27, wherein the operations further comprise

performing, based on the monitoring and the sampling, using the grid server, at least one of the following:

increasing the deduplication and replication rate, decreasing the deduplication and replication rate, increasing the ingestion rate, decreasing the ingestion rate, increasing a protection rate corresponding a combination of the deduplication rate and the replication rate, decreasing the protection rate, and/or any combination thereof.

33. The computer program product according to claim 27, wherein the sampling is performed based on at least one of the following:

periodically, automatically, and during a predetermined time interval.

34. The computer program product according to claim 27, wherein the memories of the grid server and another grid server include at least one of the following:

a non-volatile storage medium and a volatile storage medium.

35. The computer program product according to claim 27, wherein the sampling is performed based on at least one of the following parameter:

amount of data stored in a network socket receive buffer associated with the grid server, data packet round trip time associate with transmission of data between at least two grid servers, transmission control protocol advertised receive buffer size, bandwidth of a network including the grid server, a time to complete at least one disk read/write input/output request associated with the grid server, and/or any combination thereof.

36. The computer program product according to claim 35, wherein the operations further comprise

performing smoothing of the at least one parameter over a predetermined period of time;

generating, based on the smoothing, an ingestion score value;

comparing the generated ingestion score value with a threshold ingestion score value; and

performing, based on the comparing, the adjusting of the deduplication and replication rate.

37. The computer program product according to claim 27, wherein the received data is stored temporarily in a temporary storage location of the grid server.

38. The computer program product according to claim 37 wherein the operations further comprise transferring the received data from the temporary storage location of the grid server for deduplication and replication by the grid server.

39. The computer program product according to claim 37, wherein the operations further comprise performing, based on the received data temporarily stored in the temporary storage location of the grid server, at an increased speed at least one of the following:

generating at least one backup tape, generating at least one virtual machine image, performing an instant recovery of the received data from the temporary storage location, and restoring at least one file.