Scalable auxiliary copy processing using media agent resources

• US 9,898,213 B2
• Filed: 01/23/2015
• Issued: 02/20/2018
• Est. Priority Date: 01/23/2015
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Joshi, Hetalkumar N., Maranna, Chandrashekar, Vijayan, Manoj Kumar
• Primary Examiner(s): BRADLEY, MATTHEW A
• Application Number:
  US14/604,434
• Publication Number:
  US 20160216904A1
• Time in Patent Office: 1,124 Days
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0683 Plurality of storage devices

Abstract

A scalable approach is disclosed for processing auxiliary-copy jobs in a storage management system by using distributed media agent resources instead of a centralized storage manager. Enhanced media agents coordinate and control auxiliary-copy jobs and tap the storage manager to reserve data streams and provide job-specific metadata on demand. An enhanced storage manager may initially select a media agent as “coordinator” to coordinate auxiliary-copy jobs with any number of other media agents, which act as “controllers.” A coordinator media agent is generally responsible for obtaining data stream reservation information from the storage manager and assigning auxiliary-copy jobs to respective controller media agents, based on the components involved in the respective reserved data streams.

Claims

1. A method for processing auxiliary-copy jobs in a storage management system using media agent resources to offload a storage manager that centrally manages the storage management system, the method comprising:

coordinating, by a first media agent, the execution of a plurality of auxiliary-copy jobs in the storage management system,

wherein each auxiliary-copy job in the plurality of auxiliary-copy jobs comprises generating a second copy of a first copy of data generated in the storage management system, wherein the second copy is designated an auxiliary copy;

requesting, by the first media agent, from the storage manager, one or more reservations for respective one or more data streams to be used for data transfer in at least a first auxiliary-copy job in the plurality of auxiliary-copy jobs;

assigning, by the first media agent, the first auxiliary-copy job to a second media agent to execute, based at least in part on one or more data streams reserved for the first auxiliary-job by the storage manager;

executing the first auxiliary-copy job, by the second media agent, based on job-specific metadata fetched by the second media agent from the storage manager after the assigning of the first auxiliary-copy job by the first media agent to the second media agent; and

wherein the coordinating of the execution of the plurality of auxiliary-copy jobs is performed by the first media agent instead of the storage manager, and

further wherein the coordinating is based on (a) the first media agent obtaining data stream reservations from the storage manager via the requesting operation, and (b) the first media agent assigning at least one of the plurality of auxiliary-copy jobs, including the first auxiliary-copy job, to the second media agent.

2. The method of claim 1 wherein the first auxiliary copy job uses the one or more data streams reserved therefor by the storage manager in response to one or more requests from the first media agent.

3. The method of claim 1 further comprising coordinating, by the first media agent, the execution of the plurality of auxiliary-copy jobs with one or more other media agents in the storage management system, including the second media agent, wherein the coordinating comprises the first media agent assigning each respective auxiliary-copy job to one of the one or more other media agents, based on one or more data streams reserved by the storage manager for the respective auxiliary-copy job.

4. The method of claim 1 wherein the first media agent executes a coordinator process, which is triggered by the storage manager, for coordinating the plurality of auxiliary-copy jobs in the storage management system, and

wherein the second media agent executes a controller process, which is triggered by the coordinator process on the first media agent.

5. The method of claim 1 further comprising:

executing, by the first media agent, a coordinator process for coordinating the plurality of auxiliary-copy jobs that utilize one or more other media agents in the storage management system; and

causing, by the first media agent, each of the one or more other media agents, including the second media agent, to execute a respective controller process for executing one or more auxiliary-copy jobs in the plurality of auxiliary-copy jobs.

6. The method of claim 1 wherein the first media agent periodically requests the storage manager to reserve additional data streams for the first auxiliary-copy job.

7. The method of claim 1 wherein the second media agent executes a controller process for executing the first auxiliary-copy job and one or more other auxiliary-copy jobs, based at least in part on coordination supplied by the first media agent.

8. The method of claim 1 further comprising:

requesting, from the storage manager, by the first media agent, one or more reservations for respective one or more data streams to be used for data transfer in a second auxiliary-copy job in the plurality of auxiliary-copy jobs; and

executing, by the first media agent, a controller process to execute the second auxiliary-copy job, based at least in part on job-specific metadata fetched by the controller process from the storage manager and using the one or more data streams reserved for the second auxiliary-copy job by the storage manager.

9. The method of claim 1 further comprising:

coordinating, by the first media agent with a plurality of other media agents including the second media agent, the execution of the plurality of auxiliary-copy jobs including the first auxiliary-copy job, wherein the coordinating comprises the first media agent assigning each respective auxiliary-copy job to one of the media agents in the plurality of other media agents, based on one or more data streams reserved by the storage manager for the respective auxiliary-copy job.

10. The method of claim 1 wherein the first media agent coordinates the execution of the plurality of auxiliary-copy jobs in the storage management system with one or more second media agents, each of which respective second media agent executes one or more auxiliary-copy jobs assigned thereto by the first media agent;

and further comprising;

fetching from the storage manager, by the respective second media agent that is to execute a respective auxiliary-copy job, job-specific metadata about the respective auxiliary-copy job.

11. The method of claim 1 wherein the storage manager reserves one or more data streams for at least the first auxiliary-copy job in response to the requesting by the first media agent;

and

wherein the storage manager reserves additional data streams for the first auxiliary-copy job while the first auxiliary-copy job is in progress, in response to further requesting by the first media agent.

12. The method of claim 1 further comprising:

requesting, by the first media agent from the storage manager, data stream reservations for a second auxiliary-copy job, and

executing, by the first media agent, a controller process to execute the second auxiliary-copy job, based at least in part on job-specific metadata fetched by the controller process from the storage manager and using one or more data streams reserved therefor by the storage manager.

13. A method for processing auxiliary-copy jobs in a storage management system using media agent resources to offload a storage manager that manages the storage management system, the method comprising:

coordinating, by a first media agent instead of the storage manager, the execution of a plurality of auxiliary-copy jobs in the storage management system,

wherein each auxiliary-copy job in the plurality of auxiliary-copy jobs comprises generating a second copy of a first copy of data generated in the storage management system, wherein the second copy is designated an auxiliary copy;

wherein the coordinating comprises the first media agent assigning each of the plurality of auxiliary-copy jobs to a respective one of one or more other media agents in the storage management system,

wherein the assigning is based at least in part on data stream reservations made by the storage manager in response to one or more requests made thereto by the first media agent, and

wherein the assigning is further based on components involved in each respective reserved data stream, and

wherein the coordinating comprises assigning, by the first media agent, the first auxiliary-copy job to a second media agent among the one or more other media agents; and

executing the first auxiliary-copy job, by the second media agent, based on job-specific metadata fetched by the second media agent from the storage manager after the assigning of the first auxiliary-copy job by the first media agent to the second media agent.

14. The method of claim 13 wherein the first auxiliary-copy job uses one or more data streams reserved therefor by the storage manager in response to one or more requests from the first media agent.

15. The method of claim 13 further comprising:

requesting, by the first media agent, from the storage manager, one or more reservations for respective one or more data streams to be used for data transfer in at least the first auxiliary-copy job.

16. The method of claim 13 wherein the first media agent executes a coordinator process for coordinating the plurality of auxiliary-copy jobs in the storage management system.

17. The method of claim 13 wherein the second media agent executes a controller process for executing one or more auxiliary-copy jobs assigned by the first media agent.

18. The method of claim 13 wherein the first media agent executes a coordinator process for coordinating the plurality of auxiliary-copy jobs in the storage management system, and further wherein the second media agent executes a controller process for receiving auxiliary-job assignments from the coordinator process and for executing one or more auxiliary-copy jobs assigned by the coordinator process.

19. A method for processing auxiliary-copy jobs in a storage management system using media agent resources to offload a storage manager that manages the storage management system, the method comprising:

coordinating, by a first media agent, the execution of a plurality of auxiliary-copy jobs in the storage management system,

wherein each auxiliary-copy job in the plurality of auxiliary-copy jobs generates a second copy of a first copy of data generated in the storage management system, wherein the second copy is designated an auxiliary copy;

wherein the coordinating comprises the first media agent assigning each auxiliary-copy job in the plurality of auxiliary-copy jobs to a respective one of one or more other media agents in the storage management system including, assigning the first auxiliary-copy job, by the first media agent to a second media agent among the one or more other media agents,

and wherein the coordinating further comprises requesting, by the first media agent from the storage manager, one or more reservations for respective one or more data streams to be used for data transfer in at least the first auxiliary-copy job, and

wherein the assigning of the first auxiliary-copy job by the first media agent to the second media agent is based on resources for reserved one or more data streams for the first auxiliary-copy job;

fetching, by the second media agent, job-specific metadata about the first auxiliary-copy job from the storage manager in the storage management system; and

executing the first auxiliary-copy job, by the second media agent, based on the job-specific metadata fetched from the storage manager.

20. The method of claim 19 further comprising:

requesting, by the first media agent during the course of the first auxiliary-copy job, one or more reservations for additional respective data streams to be used for data transfer for the first auxiliary-copy job, wherein the auxiliary-copy job continues executing based on the additional respective data streams.