Scalable auxiliary copy processing using media agent resources

• US 10,168,930 B2
• Filed: 12/19/2017
• Issued: 01/01/2019
• Est. Priority Date: 01/23/2015
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Joshi, Hetalkumar N., Maranna, Chandrashekar, Vijayan, Manoj Kumar
• Primary Examiner(s): Bradley, Matthew
• Application Number:
  US15/847,781
• Publication Number:
  US 20180113637A1
• Time in Patent Office: 378 Days
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0683 Plurality of storage devices

Abstract

A scalable approach is disclosed for processing auxiliary-copy jobs in a storage management system by using distributed media agent resources instead of a centralized storage manager. Enhanced media agents coordinate and control auxiliary-copy jobs and tap the storage manager to reserve data streams and provide job-specific metadata on demand. An enhanced storage manager may initially select a media agent as “coordinator” to coordinate auxiliary-copy jobs with any number of other media agents, which act as “controllers.” A coordinator media agent is generally responsible for obtaining data stream reservation information from the storage manager and assigning auxiliary-copy jobs to respective controller media agents, based on the components involved in the respective reserved data streams.

Claims

1. A method for using a media agent to offload a storage manager that manages a data storage management system, the method comprising:

coordinating, by a first media agent, execution of one or more auxiliary-copy jobs in the data storage management system,

wherein each auxiliary-copy job comprises generating a second copy of a first copy of data generated in the data storage management system;

requesting, by the first media agent, that the storage manager reserve one or more data streams for a first auxiliary-copy job;

assigning execution of the first auxiliary-copy job, by the first media agent to a second media agent in the data storage management system,

based at least in part on the one or more data streams reserved for the first auxiliary-copy job by the storage manager;

after the assigning, fetching by the second media agent from the storage manager metadata for the first auxiliary-copy job;

executing the first auxiliary-copy job by the second media agent, based on the metadata fetched from the storage manager, and using the one or more reserved data streams; and

wherein the coordinating of the execution of the one or more auxiliary-copy jobs in the data storage management system is performed by the first media agent instead of the storage manager.

2. The method of claim 1 wherein the assigning the execution of the first auxiliary-copy job by the first media agent to the second media agent is further based on components in the data storage management system that are involved in each of the one or more data streams reserved by the storage manager.

3. The method of claim 1 wherein the metadata fetched by the second media agent is specific to the first auxiliary-copy job.

4. The method of claim 1 wherein the coordinating of the execution of the one or more auxiliary-copy jobs comprises the first media agent assigning a given auxiliary-copy job to one of one or more other media agents in the data storage management system.

5. The method of claim 1 wherein the coordinating of the execution of the one or more auxiliary-copy jobs comprises the first media agent assigning a given auxiliary-copy job to one of one or more other media agents in the data storage management system, based on one or more data streams reserved by the storage manager for the given auxiliary-copy job.

6. The method of claim 1 wherein the first media agent executes a coordinator process, which is triggered by the storage manager, for coordinating the one or more auxiliary-copy jobs in the data storage management system, and

wherein the second media agent executes a controller process, which is triggered by the coordinator process on the first media agent.

7. The method of claim 1 further comprising:

executing, by the first media agent, a coordinator process for coordinating the one or more auxiliary-copy jobs that utilize one or more other media agents in the data storage management system; and

causing, by the first media agent, each of the one or more other media agents to execute a respective controller process for executing respective at least one of the one or more auxiliary-copy jobs.

8. The method of claim 1 wherein the first media agent periodically requests that the storage manager reserve additional data streams for the first auxiliary-copy job.

9. The method of claim 1 wherein the second media agent executes a controller process for executing the first auxiliary-copy job and one or more other auxiliary-copy jobs, based at least in part on coordination supplied by the first media agent.

10. The method of claim 1 further comprising:

requesting, by the first media agent, that the storage manager reserve one or more data streams to be used in a second auxiliary-copy job; and

executing, by the first media agent, a controller process for executing the second auxiliary-copy job by the first media agent,

based at least in part on metadata fetched by the controller process from the storage manager and using the one or more data streams reserved for the second auxiliary-copy job by the storage manager.

11. The method of claim 1 further comprising:

further requesting in the course of execution of the first auxiliary-copy job, by the first media agent, that the storage manager reserve one or more additional data streams for the first auxiliary-copy job.

12. A data storage management system comprising:

a storage manager that executes on a computing device comprising one or more processors and computer memory;

a first media agent that executes on a computing device comprising one or more processors and computer memory;

a second media agent that executes on a computing device comprising one or more processors and computer memory;

wherein the first media agent is configured to;

coordinate execution of one or more auxiliary-copy jobs in the data storage management system,

wherein each auxiliary-copy job comprises generating a second copy of a first copy of data generated in the data storage management system,

request that the storage manager reserve one or more data streams for at least a first auxiliary-copy job, and

assign execution of the first auxiliary-copy job to the second media agent,

based at least in part on the one or more data streams reserved for the first auxiliary-copy job by the storage manager;

wherein the second media agent is configured to;

execute the first auxiliary-copy job using the one or more reserved data streams, and based on job-specific metadata for the first auxiliary-copy job fetched from the storage manager after being assigned the execution of the first auxiliary-copy job; and

wherein the first media agent is configured to coordinate, instead of the storage manager, the execution of the one or more auxiliary-copy jobs in the data storage management system.

13. The data storage management system of claim 12 wherein the first media agent is configured to assign the execution of the first auxiliary-copy job to the second media agent further based on components involved in each of the one or more data streams reserved by the storage manager for the first auxiliary-copy job.

14. The data storage management system of claim 12 further comprising other media agents in addition to the second media agent, each respective other media agent executing on a respective computing device having one or more processors and computer memory;

and

wherein to coordinate the execution of the one or more auxiliary-copy jobs, the first media agent is further configured to assign a given auxiliary-copy job to one of one or more other media agents in the data storage management system.

15. The data storage management system of claim 12 wherein the first media agent is further configured to execute a coordinator process for coordinating the one or more auxiliary-copy jobs in the data storage management system;

wherein the storage manager is configured to trigger the coordinator process at the first media agent;

wherein the second media agent is further configured to execute a controller process; and

wherein the first media agent is further configured to trigger the controller process at the second media agent.

16. The data storage management system of claim 12 wherein the first media agent is further configured to periodically request that the storage manager reserve additional data streams for the first auxiliary-copy job.

17. The data storage management system of claim 12 wherein the first media agent is further configured to:

request that the storage manager reserve one or more data streams to be used in a second auxiliary-copy job; and

execute a controller process for executing the second auxiliary-copy job, based at least in part on metadata fetched by the controller process from the storage manager and using the one or more data streams reserved by the storage manager for the second auxiliary-copy job.

18. A computer-readable medium, excluding transitory propagating signals, storing instructions that, when executed by a computing device comprising one or more processors and computer memory, cause the computing device to perform operations comprising:

coordinating execution of one or more auxiliary-copy jobs in a data storage management system managed by a storage manager,

wherein each auxiliary-copy job comprises generating a second copy of a first copy of data generated in the data storage management system;

requesting that the storage manager reserve one or more data streams for at least a first auxiliary-copy job;

assigning execution of the first auxiliary-copy job to another computing device in the data storage management system,

based at least in part on the one or more data streams reserved for the first auxiliary-copy job by the storage manager;

requesting that the storage manager reserve one or more data streams to be used in a second auxiliary-copy job;

executing the second auxiliary-copy job, based at least in part on metadata fetched from the storage manager and using the one or more data streams reserved for the second auxiliary-copy job by the storage manager; and

wherein the coordinating of the execution of the one or more auxiliary-copy jobs in the data storage management system is performed by the computing device instead of the storage manager.

19. The computer-readable medium of claim 18 wherein the coordinating is performed by a coordinator process executing on the computing device and triggered by the storage manager.

20. The computer-readable medium of claim 18 wherein the coordinating is performed by a coordinator process executing on the computing device and triggered by the storage manager;

and

wherein the executing of the second auxiliary-copy job is performed by a controller process executing on the computing device and triggered by the coordinator process.