Scalable auxiliary copy processing in a storage management system using media agent resources

• US 9,904,481 B2
• Filed: 01/23/2015
• Issued: 02/27/2018
• Est. Priority Date: 01/23/2015
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Joshi, Hetalkumar N., Maranna, Chandrashekar, Vijayan, Manoj Kumar
• Primary Examiner(s): BRADLEY, MATTHEW A
• Application Number:
  US14/604,431
• Publication Number:
  US 20160216903A1
• Time in Patent Office: 1,131 Days
• Field of Search: None
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0683 Plurality of storage devices

Abstract

A scalable approach is disclosed for processing auxiliary-copy jobs in a storage management system by using distributed media agent resources instead of a centralized storage manager. Enhanced media agents coordinate and control auxiliary-copy jobs and tap the storage manager to reserve data streams and provide job-specific metadata on demand. An enhanced storage manager may initially select a media agent as “coordinator” to coordinate auxiliary-copy jobs with any number of other media agents, which act as “controllers.” A coordinator media agent is generally responsible for obtaining data stream reservation information from the storage manager and assigning auxiliary-copy jobs to respective controller media agents, based on the components involved in the respective reserved data streams.

Claims

1. A storage management system for auxiliary-copy processing using media agent resources to offload a storage manager of the storage management system, the storage management system comprising:

the storage manager for reserving one or more data streams for transfer of data in a first auxiliary-copy job from a source to a destination in the storage management system,

wherein the source comprises a first copy of data generated in the storage management system, and

further wherein the destination stores a second copy of the first copy, which second copy of the first copy is generated during the first auxiliary-copy job and is designated an auxiliary copy;

a first media agent in communication with the storage manager;

a second media agent in communication with the storage manager and with the first media agent;

wherein the first media agent is configured to;

request, from the storage manager, data stream reservations for the first auxiliary-copy job, and

based on components involved in the data stream reservations, coordinate with the second media agent an execution of the first auxiliary-copy job via the one or more data streams reserved by the storage manager;

wherein the second media agent is configured to;

fetch, from the storage manager, job-specific metadata about the first auxiliary-copy job,

generate the auxiliary copy, based at least in part on the job-specific metadata fetched from the storage manager; and

wherein the storage manager is configured to select the first media agent to coordinate, instead of the storage manager, auxiliary-copy jobs in the storage management system.

2. The storage management system of claim 1 wherein the second media agent is further configured to report status about the first auxiliary-copy job to the first media agent.

3. The storage management system of claim 1 wherein the storage manager is configured to prioritize the first auxiliary-copy job within a plurality of storage management jobs pending in the storage management system.

4. The storage management system of claim 1 wherein the storage manager is configured to reserve one or more data streams on demand for one or more auxiliary-copy jobs, based on one or more requests from the first media agent.

5. The storage management system of claim 1 wherein the storage manager is configured to provide job-specific metadata on demand, based on one or more requests from respective second media agents.

6. The storage management system of claim 1 wherein the storage manager is configured to cause the first media agent to execute a coordinator process for coordinating, with one or more second media agents in the storage management system, a respective execution of a plurality of auxiliary-copy jobs.

7. The storage management system of claim 1 wherein the first media agent is further configured to execute a coordinator process for coordinating a plurality of auxiliary-copy jobs that utilize one or more second media agents in the storage management system.

8. The storage management system of claim 1 wherein the first media agent is further configured to:

execute a coordinator process for coordinating a plurality of auxiliary-copy jobs that utilize one or more second media agents in the storage management system, and

cause each of the one or more second media agents to execute a respective controller process for executing respective one or more auxiliary-copy jobs in the plurality of auxiliary-copy jobs, based at least in part on coordination supplied by the coordinator process of the first media agent.

9. The storage management system of claim 1 wherein the first media agent is further configured to periodically request the storage manager to reserve additional data streams for the first auxiliary-copy job.

10. The storage management system of claim 1 wherein the second media agent is further configured to execute a controller process for executing the first auxiliary-copy job and one or more other auxiliary-copy jobs, based at least in part on coordination supplied by the first media agent.

11. The storage management system of claim 1 further comprising:

a third media agent in communication with the storage manager and with the first media agent;

wherein the first media agent is further configured to;

request, from the storage manager, data stream reservations for a second auxiliary-copy job, and

coordinate with the third media agent to execute the second auxiliary-copy job via one or more data streams reserved by the storage manager for the second auxiliary-copy job; and

wherein the third media agent is configured to execute the second auxiliary-copy job, based at least in part on job-specific metadata fetched from the storage manager.

12. A method for auxiliary-copy processing in a storage management system using media agent resources to offload a storage manager that manages the storage management system, the method comprising:

causing, by the storage manager that manages the storage management system, a first media agent to coordinate execution of a plurality of auxiliary-copy jobs in the storage management system,

wherein each auxiliary-job in the plurality of auxiliary-copy jobs comprises copying a respective first copy of data generated in the storage management system to a respective second copy, which is designated an auxiliary copy;

requesting from the storage manager, by the first media agent, one or more reservations for respective one or more data streams to be used for data transfer in at least a first auxiliary-copy job among the plurality of auxiliary-copy jobs coordinated by the first media agent;

assigning, by the first media agent instead of the storage manager, a second media agent to execute the first auxiliary-copy job,

based at least in part on components involved in data streams reserved by the storage manager for the first auxiliary-copy job; and

executing the first auxiliary-copy job, by the second media agent as assigned by the first media agent,

based on job-specific metadata about the first auxiliary-copy job fetched by the second media agent from the storage manager,

wherein the first auxiliary-copy job uses one or more data streams reserved therefor by the storage manager.

13. The method of claim 12 wherein, for each respective auxiliary-copy job, the storage manager reserves one or more data streams in response to the first media agent so requesting.

14. The method of claim 12 wherein, the storage manager reserves one or more data streams in response to periodic requests received from the first media agent.

15. The method of claim 12 wherein the storage manager reserves one or more data streams for at least the first auxiliary-copy job in response to the requesting by the first media agent;

and

wherein the storage manager reserves additional data streams for the first auxiliary-copy job while the first auxiliary-copy job is in progress, in response to further requesting by the first media agent.

16. The method of claim 12 further comprising:

requesting, by the first media agent from the storage manager data stream reservations for a second auxiliary-copy job, and

executing, by the first media agent, a controller process to execute the second auxiliary-copy job, based at least in part on job-specific metadata fetched by the controller process from the storage manager and using one or more data streams reserved therefor by the storage manager.

17. A storage management system for processing auxiliary-copy jobs using media agent resources instead of a storage manager for managing the storage management system, the storage management system comprising:

the storage manager for managing the storage management system;

a first media agent in communication with the storage manager;

a second media agent in communication with the storage manager and with the first media agent;

wherein the first media agent is configured, based on being designated a coordinator by the storage manager, to;

coordinate the execution of a plurality of auxiliary-copy jobs in the storage management system, wherein each auxiliary-copy job in the plurality of auxiliary-copy jobs comprises generating a second copy of a first copy of data generated in the storage management system, wherein the second copy is designated an auxiliary copy,

request from the storage manager one or more reservations for respective one or more data streams to be used for data transfer in at least a first auxiliary-copy job in the plurality of auxiliary-copy jobs, and

assign the first auxiliary-copy job to the second media agent to execute, based at least in part on one or more data streams reserved for the first auxiliary-job by the storage manager;

wherein the second media agent is configured, based on being assigned by the first media agent, to;

execute the first auxiliary-copy job, based on job-specific metadata fetched by the second media agent from the storage manager.

18. The storage management system of claim 17 wherein the first auxiliary copy job uses the one or more data streams reserved therefor by the storage manager in response to one or more requests from the first media agent.

19. The storage management system of claim 17 wherein the first media agent is further configured to:

coordinate the execution of the plurality of auxiliary-copy jobs with one or more media agents in the storage management system, including the second media agent, wherein the coordinating comprises the first media agent assigning each respective auxiliary-copy job to one of the one or more media agents, based on one or more data streams reserved by the storage manager for the respective auxiliary-copy job.

20. The storage management system of claim 17 wherein the second media agent is further configured to execute a controller process for executing the first auxiliary-copy job and one or more other auxiliary-copy jobs, based at least in part on coordination supplied by the first media agent.