Intelligent local management of data stream throttling in secondary-copy operations

• US 10,313,243 B2
• Filed: 02/24/2015
• Issued: 06/04/2019
• Est. Priority Date: 02/24/2015
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Thanasekaran, Vijayan
• Primary Examiner(s): Park, Jeong S
• Application Number:
  US14/630,139
• Publication Number:
  US 20160248676A1
• Time in Patent Office: 1,561 Days
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0646 Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 47/12 Avoiding congestion; Recovering from congestion
• CPC Class Codes: H04L 47/522 Dynamic queue service slot or variable bandwidth allocation
• CPC Class Codes: H04L 47/781 Centralised allocation of resources
• CPC Class Codes: H04L 47/83 based on usage prediction

Abstract

Intelligent local management of data stream throttling in data movement operations, such as secondary-copy operations in a storage management system, is disclosed. A local throttling manager may intelligently interoperate with co-resident data agents and/or a media agent executing on any given local computing device, whether a client computing device or a secondary storage computing device. The local throttling manager may allocate and manage the available bandwidth for various jobs and their constituent data streams—across the data agents and/or media agent. Bandwidth is dynamically allocated and re-allocated to data streams used by ongoing jobs, in response to new jobs starting and old jobs completing, without having to pause and restart ongoing jobs to accommodate bandwidth adjustments. The illustrative embodiment also provides local users with a measure of control over data streams—to suspend, pause, and/or resume them—independently from the centralized storage manager that manages the storage management system as a whole.

Claims

1. A method for managing data streams that originate from a client computing device in a storage management system, the method comprising:

managing, by a throttling manager that executes on the client computing device, a bandwidth available for data streams that are registered at and originate from the client computing device, wherein the managing comprises;

receiving, by the throttling manager, one or more respective registrations for one or more first data streams for a first secondary-copy job for first primary data accessible to the client computing device,

wherein a first data agent that also executes on the client computing device (I) submits the one or more respective registrations for the first data streams, and (II) uses the first data streams to transfer the first primary data to a first media agent that executes on a secondary storage computing device that is distinct from the client computing device;

receiving, by the throttling manager, one or more respective registrations for one or more second data streams for a second secondary-copy job for second primary data accessible to the client computing device, wherein a second data agent that also executes on the client computing device (A) submits the one or more respective registrations for the second data streams, and (B) uses the second data streams to transfer the second primary data to the first media agent;

allocating, by the throttling manager, based on the bandwidth available for data streams that are registered at and originate from the client computing device (i) first respective bandwidth values for the first data streams to be used by the first data agent in the first secondary-copy job and (ii) second respective bandwidth values for the second data streams to be used by the second data agent in the second secondary-copy job; and

after the second secondary-copy job completes, (a) receiving, by the throttling manager from the second data agent, one or more respective de-registrations for the second data streams used in the second secondary-copy job and (b) updating, by the throttling manager, based on the de-registrations, the first respective bandwidth values to third respective bandwidth values for the first data streams in use by the first data agent for the first secondary-copy job,

wherein the one or more first data streams continue operating at the updated third respective bandwidth values which differ from the first respective bandwidth values.

2. The method of claim 1 wherein the managing further comprises:

receiving, by the throttling manager, one or more respective registrations for one or more third data streams for a third secondary-copy job for third primary data accessible to the client computing device, wherein a third data agent that also executes on the client computing device a. submits the one or more respective registrations for the third data streams, and b. uses the third data streams to transfer the third primary data to the first media agent; and

based on the bandwidth available for data streams that are registered at and originate from the client computing device (i) allocating, by the throttling manager, fourth respective bandwidth values for the third data streams, and (ii) updating, by the throttling manager, the third respective bandwidth values to fifth respective bandwidth values for the first data streams in use by the first secondary-copy job, wherein the one or more first data streams continue operating at the updated fifth respective bandwidth values which differ from the third respective bandwidth values.

3. The method of claim 1 wherein the managing comprises maximizing, by the throttling manager, based on the bandwidth available for data streams that are registered at and originate from the client computing device, the respective bandwidth values allocated and re-allocated to registered data streams.

4. The method of claim 1 wherein the throttling manager allocates and re-allocates bandwidth values to respective data streams registered by respective data agents that execute on the client computing device so that a total bandwidth used by the registered data streams is maximized relative to the bandwidth available for data streams that are registered at and originate from the client computing device.

5. The method of claim 1 further comprising:

receiving, by the throttling manager from the first data agent, during the first secondary-copy job, a value for an amount of data transferred so far by the first secondary-copy job.

6. At least one computer-readable medium, excluding transitory propagating signals, storing instructions that, when executed by a computing device comprising one or more processors and computer memory, perform a method for management of data stream throttling in a storage management system, the method comprising:

receiving, by a first data agent that executes on the computing device, instructions for a first copy job for first primary data associated with the first data agent,

wherein the instructions comprise one or more parameters for one or more first data streams that are to be used by the first data agent for transferring the first primary data to a first media agent during the first copy job;

transmitting, by the first data agent, to a throttling manager, one or more respective registrations for the one or more first data streams,

wherein the throttling manager also executes on the computing device;

receiving, by the first data agent, from the throttling manager, respective first bandwidth values for the one or more first data streams;

executing the first copy job, at least in part by the first data agent,

wherein the first copy job comprises transferring at least part of the first primary data to the first media agent via the one or more first data streams at the respective first bandwidth values received from the throttling manager, and

wherein the one or more first data streams originate from the first client computing device;

receiving, by the first data agent, from the throttling manager, respective second bandwidth values for the one or more first data streams used by the first copy job;

continuing the one or more first data streams at the respective second bandwidth values received from the throttling manager, wherein the respective second bandwidth values differ from the respective first bandwidth values;

after the first copy job is completed, transmitting, by the first data agent, to the throttling manager, respective de-registrations for the one or more first data streams; and

wherein the first bandwidth values are changed to the second bandwidth values, based on a second copy job executed at least in part by a second data agent that also executes on the computing device,

wherein the second copy job utilizes one or more second data streams that are registered with the throttling manager, and

wherein the one or more second data streams used by the second copy job also originate at the first client computing device and are to be used for transferring second primary data to the first media agent during the second copy job.

7. The computer-readable medium of claim 6 further comprising:

transmitting, by the second data agent that also executes on the first client computing device, to the throttling manager, one or more respective registrations for the one or more second data streams;

receiving, by the second data agent, from the throttling manager, respective third bandwidth values for the one or more second data streams;

executing the second copy job, at least in part by the second data agent, wherein the second copy job comprises transferring at least part of the second primary data to the first media agent via the one or more second data streams at the respective third bandwidth values received from the throttling manager; and

wherein the second bandwidth values and the third bandwidth values are based on the first data streams and the second data streams being registered with the throttling manager.

8. The computer-readable medium of claim 6 wherein the second bandwidth values are different from the first bandwidth values, based on a maximum bandwidth that is managed by the throttling manager for data streams registered at and originating from the first client computing device and terminating at the first media agent.

9. The computer-readable medium of claim 6 wherein the instructions are received by the first data agent from a storage manager that manages the first copy job and the second copy job in the storage management system.

10. The computer-readable medium of claim 6 further comprising,

transmitting, by the first data agent to the throttling manager, in the course of the first copy job, a value for an amount of data transferred so far by the first copy job.