Dynamic throughput ingestion of backup sources

• US 10,298,680 B1
• Filed: 09/23/2015
• Issued: 05/21/2019
• Est. Priority Date: 09/23/2015
• Status: Active Grant

• Current Assignee: Cohesity, Inc.
• Original Assignee: Cohesity, Inc.
• Inventors: Thomas, Markose, Manjunath, Chinmaya
• Primary Examiner(s): Vostal, Ondrej C
• Application Number:
  US14/863,178
• Time in Patent Office: 1,336 Days
• Field of Search: 709219
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 43/0888 Throughput
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/51 Discovery or management thereof, e.g. service location protocol [SLP] or web services

Abstract

A method and apparatus for dynamically adjusting an ingestion rate for backup operations on a source system. The method generally includes monitoring a resource utilization related to one or more performance metrics of the source system in performing at least a primary workload. Based on the monitored resource utilization, the backup system determines a data ingestion rate for backup operations on the source system. The backup system ingests data from the source system to a backup repository at the determined data ingestion rate.

Claims

1. A method for dynamically adjusting an ingestion rate for backup operations on a source system, comprising:

monitoring, by a backup system, a resource utilization related to one or more performance metrics of the source system in performing at least a primary workload, wherein the backup system is configured to query the source system to obtain the resource utilization related to one or more performance metrics of the source system in performing at least the primary workload;

based on the monitored resource utilization, determining, by the backup system, a data ingestion rate for backup operations on the source system;

adjusting the data ingestion rate from the source system, wherein adjusting the data ingestion rate from the source system comprises adjusting an amount of resources allocated to the primary workload at the source system by an equal amount to resources allocated to the backup operations on the source system;

ingesting, at the backup system, data from the source system at the determined data ingestion rate; and

writing, by the backup system, the ingested data to a distributed file system at a storage repository, wherein the distributed file system implements a directory structure in which the ingested data is stored, wherein the storage repository is separate from the backup system.

2. The method of claim 1, further comprising:

comparing the monitored resource utilization to a threshold; and

if the monitored resource utilization exceeds the threshold, decreasing the data ingestion rate such that the backup operations do not interfere with the primary workload.

3. The method of claim 1, further comprising:

comparing the monitored resource utilization to a threshold; and

if the monitored resource utilization is less than the threshold, increasing the data ingestion rate to accelerate the backup operations.

4. The method of claim 1, wherein the monitored resource utilization comprises activity at a network interface of the source system.

5. The method of claim 1, wherein the monitored resource utilization comprises processor idle time at the source system.

6. The method of claim 1, wherein the monitored resource utilization comprises storage subsystem throughput at the source system.

7. The method of claim 1, further comprising adjusting the data ingestion rate from the source system at least in part by adjusting an amount of resources allocated to a primary workload at the source system by a greater amount than an adjustment made to resources allocated to backup operations on the source system.

8. The method of claim 1, further comprising adjusting the data ingestion rate from the source system at least in part by adjusting an amount of resources allocated to backup operations on the source system by a greater amount than an adjustment made to resources allocated to a primary workload at the source system.

9. The method of claim 1, further comprising comparing the ingested data to data previously stored at the storage repository, wherein the ingested data is written from the backup system to the storage repository based on a comparison of the ingested data to the data previously stored at the storage repository.

10. The method of claim 9, wherein the backup system is configured to write a pointer to the data previously stored at the storage repository in the event the comparison indicates the ingested data is a same data as the data previously stored at the storage repository.

11. The method of claim 9, wherein the backup system is configured to write the ingested data to the storage repository in the event the comparison indicates the ingested data is a different data than the data previously stored at the storage repository.

12. A method for dynamically adjusting an ingestion rate for backup operations on a source system, comprising:

monitoring, by a backup system, a resource utilization related to one or more performance metrics of the source system in performing at least a primary workload, wherein the backup system is configured to query the source system to obtain the resource utilization related to one or more performance metrics of the source system in performing at least the primary workload, wherein the monitored resource utilization comprises storage system latency at the source system;

based on the monitored resource utilization, determining, by the backup system, a data ingestion rate for backup operations on the source system;

ingesting, at the backup system, data from the source system at the determined data ingestion rate; and

writing, by the backup system, the ingested data to a distributed file system at a storage repository, wherein the distributed file system implements a directory structure in which the ingested data is stored, wherein the storage repository is separate from the backup system.

13. A non-transitory computer readable medium comprising instructions that, when executed, perform an operation for dynamically adjusting an ingestion rate for backup operations on a source system, the operations comprising:

monitoring, by a backup system, a resource utilization related to one or more performance metrics of the source system in performing at least a primary workload, wherein the backup system is configured to query the source system to obtain the resource utilization related to one or more performance metrics of the source system in performing at least the primary workload;

based on the monitored resource utilization, determining, by the backup system, a data ingestion rate for backup operations on the source system;

adjusting the data ingestion rate from the source system, wherein adjusting the data ingestion rate from the source system comprises adjusting an amount of resources allocated to the primary workload at the source system by an equal amount to resources allocated to the backup operations on the source system;

ingesting, at the backup system, data from the source system at the determined data ingestion rate; and

writing, by the backup system, the ingested data to a distributed file system at a storage repository, wherein the distributed file system implements a directory structure in which the ingested data is stored, wherein the storage repository is separate from the backup system.

14. The non-transitory computer readable medium of claim 13, further comprising:

comparing the monitored resource utilization to a threshold; and

if the monitored resource utilization exceeds the threshold, decreasing the data ingestion rate such that the backup operations do not interfere with the primary workload.

15. The non-transitory computer readable medium of claim 13, further comprising:

comparing the monitored resource utilization to a threshold; and

if the monitored resource utilization is less than the threshold, increasing the data ingestion rate to accelerate the backup operations.

16. The non-transitory computer readable medium of claim 13, wherein the monitored resource utilization comprises at least one of:

activity at a network interface of the source system;

processor idle time at the source system;

storage subsystem throughput at the source system;

or

storage subsystem latency at the source system.

17. A system comprising:

a processor; and

a memory storing one or more instructions, which, when executed by the processor, performs an operation for dynamically adjusting an ingestion rate for backup operations on a source system, the operation comprising;

monitoring, by a backup system, a resource utilization related to one or more performance metrics of the source system in performing at least a primary workload, wherein the backup system is configured to query the source system to obtain the resource utilization related to one or more performance metrics of the source system in performing at least the primary workload;

based on the monitored resource utilization, determining, by a backup system, a data ingestion rate for backup operations on the source system;

adjust the data ingestion rate from the source system, wherein adjusting the data ingestion rate from the source system comprises adjusting an amount of resources allocated to the primary workload at the source system by an equal amount to resources allocated to the backup operations on the source system;

ingesting, at the backup system, data from the source system at the determined data ingestion rate; and

writing, by the backup system, the ingested data to a distributed file system at a storage repository, wherein the distributed file system implements a directory structure in which the ingested data is stored, wherein the storage repository is separate from the backup system.

18. The system of claim 17, further comprising:

comparing the monitored resource utilization to a threshold; and

if the monitored resource utilization exceeds the threshold, decreasing the data ingestion rate such that the backup operations do not interfere with the primary workload.

19. The system of claim 17, further comprising:

comparing the monitored resource utilization to a threshold; and

if the monitored resource utilization is less than the threshold, increasing the data ingestion rate to accelerate the backup operations.

20. The system of claim 17, wherein the monitored resource utilization comprises at least one of:

activity at a network interface of the source system;

processor idle time at the source system;

storage subsystem throughput at the source system;

or

storage subsystem latency at the source system.

21. The system of claim 17, further comprising adjusting the data ingestion rate from the source system at least in part by adjusting an amount of resources allocated to a primary workload at the source system by a greater amount than an adjustment made to resources allocated to backup operations on the source system.

22. The system of claim 17, further comprising adjusting the data ingestion rate from the source system at least in part by adjusting an amount of resources allocated to backup operations on the source system by a greater amount than an adjustment made to resources allocated to a primary workload at the source system.