Scalable auxiliary copy processing in a data storage management system using media agent resources

• US 10,168,931 B2
• Filed: 12/21/2017
• Issued: 01/01/2019
• Est. Priority Date: 01/23/2015
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Joshi, Hetalkumar N., Maranna, Chandrashekar, Vijayan, Manoj Kumar
• Primary Examiner(s): Bradley, Matthew
• Application Number:
  US15/851,539
• Publication Number:
  US 20180121115A1
• Time in Patent Office: 376 Days
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0683 Plurality of storage devices

Abstract

A scalable approach is disclosed for processing auxiliary-copy jobs in a storage management system by using distributed media agent resources instead of a centralized storage manager. Enhanced media agents coordinate and control auxiliary-copy jobs and tap the storage manager to reserve data streams and provide job-specific metadata on demand. An enhanced storage manager may initially select a media agent as “coordinator” to coordinate auxiliary-copy jobs with any number of other media agents, which act as “controllers.” A coordinator media agent is generally responsible for obtaining data stream reservation information from the storage manager and assigning auxiliary-copy jobs to respective controller media agents, based on the components involved in the respective reserved data streams.

Claims

1. A non-transitory computer-readable medium storing instructions that, when executed by a computing device that comprises one or more processors and computer memory, cause the computing device to perform operations comprising:

causing a first media agent component of a data storage management system, to coordinate a first auxiliary-copy job from a source to a destination in the data storage management system,

wherein the source comprises a first copy of data generated in the data storage management system, and

wherein the destination stores a second copy of the first copy, wherein the second copy is generated during the first auxiliary-copy job;

receiving, from the first media agent, reservation requests for data streams needed for the first auxiliary-copy job;

reserving, according to the reservation requests, the data streams for transferring data in the first auxiliary-copy job;

providing to the first media agent information about components involved in the reserved data streams, to enable the first media agent to determine a second media agent suitable for executing the first auxiliary-copy job; and

providing job-specific metadata about the first auxiliary-copy job to the second media agent for executing the first auxiliary-copy job.

2. The non-transitory computer-readable medium of claim 1, wherein the operations further comprise:

prioritizing the first auxiliary-copy job within a plurality of storage management jobs pending in the data storage management system.

3. The non-transitory computer-readable medium of claim 1, wherein by causing the first media agent to coordinate the auxiliary-copy job in the data storage management system, the computing device is off-loaded from having to coordinate the first auxiliary-copy job.

4. The non-transitory computer-readable medium of claim 1, wherein the providing of job-specific metadata about the first auxiliary-copy job to the second media agent is performed in response to a fetching operation initiated by the second media agent.

5. The non-transitory computer-readable medium of claim 1, wherein the operations further comprise:

causing the first media agent to coordinate a plurality of auxiliary-copy jobs in the data storage management system, including the first auxiliary-copy job, thereby off-loading the first computing device from having to coordinate the plurality of auxiliary-copy jobs.

6. The non-transitory computer-readable medium of claim 1, wherein the operations further comprise:

triggering a coordinator process to execute at the first media agent, thereby causing the first media agent to coordinate the auxiliary-copy job.

7. A data storage management system comprising:

a first media agent that executes on a first computing device comprising one or more processors and computer memory;

a second media agent that executes on a second computing device comprising one or more processors and computer memory;

a storage manager that executes on a third computing device comprising one or more processors and computer memory;

wherein the storage manager is configured to;

cause the first media agent to coordinate a first auxiliary-copy job from a source to a destination in the data storage management system,

wherein the source comprises a first copy of data generated in the data storage management system, and

wherein the destination stores a second copy of the first copy, wherein the second copy is generated during the first auxiliary-copy job;

receive, from the first media agent, reservation requests for data streams needed for the first auxiliary-copy job;

reserve, according to the reservation requests, the data streams for transferring data in the first auxiliary-copy job;

transmit to the first media agent information about components of the data storage management system that are involved in the reserved data streams, to enable the first media agent to determine that the second media agent is suitable for executing the first auxiliary-copy job; and

transmit job-specific metadata about the first auxiliary-copy job to the second media agent for executing the first auxiliary-copy job.

8. The data storage management system of claim 7, wherein by being configured to cause the first media agent to coordinate the first auxiliary-copy job in the data storage management system, the storage manager is off-loaded from having to coordinate the first auxiliary-copy job.

9. The data storage management system of claim 7, wherein the storage manager is further configured to:

cause the first media agent to coordinate a plurality of auxiliary-copy jobs in the data storage management system, including the first auxiliary-copy job, thereby off-loading the storage manager from having to coordinate the plurality of auxiliary-copy jobs.

10. The data storage management system of claim 7, wherein the storage manager is further configured to:

trigger a coordinator process at the first media agent, thereby causing the first media agent to coordinate the first auxiliary-copy job.

11. A method for auxiliary-copy processing using media agent resources to offload a storage manager in a data storage management system, the method comprising:

selecting, by the storage manager, a first media agent in the data storage management system, to coordinate, instead of the storage manager, at least one auxiliary-copy job in the data storage management system;

receiving, by the storage manager from the first media agent, one or more requests for data stream reservations for a first auxiliary-copy job from a source to a destination in the data storage management system,

wherein the source comprises a first copy of data generated in the data storage management system, and

wherein the destination stores a second copy of the first copy, wherein the second copy is generated during the first auxiliary-copy job;

reserving, by the storage manager, according to the one or more requests, one or more data streams for transferring data in the first auxiliary-copy job;

based on components involved in the one or more data streams reserved by the storage manager, coordinating, by the first media agent an execution of the first auxiliary-copy job, which uses the one or more data streams and a second media agent;

fetching, by the second media agent, from the storage manager, metadata about the first auxiliary-copy job; and

generating the second copy of the first copy as part of the execution of the first auxiliary-job, by the second media agent, based at least in part on the metadata fetched from the storage manager, and further based on the coordinating by the first media agent.

12. The method of claim 11 further comprising:

reporting, by the second media agent to the first media agent, status about the first auxiliary-copy job.

13. The method of claim 11 further comprising:

prioritizing, by the storage manager, the first auxiliary-copy job within a plurality of storage management jobs pending in the data storage management system.

14. The method of claim 11 further comprising:

reserving, by the storage manager one or more data streams on demand, for one or more auxiliary-copy jobs, based on one or more requests received from the first media agent.

15. The method of claim 11 further comprising:

providing, by the storage manager on demand, the metadata about the first auxiliary-copy job.

16. The method of claim 11 further comprising:

causing, by the storage manager, the first media agent to execute a coordinator process for coordinating the at least one auxiliary-copy job in the data storage management system.

17. The method of claim 11 further comprising:

executing, by the first media agent, a coordinator process for coordinating a plurality of auxiliary-copy jobs that utilize one or more second media agents in the storage management system.

18. The method of claim 11 further comprising:

executing, by the first media agent, a coordinator process for coordinating a plurality of auxiliary-copy jobs that utilize one or more second media agents in the data storage management system; and

causing, by the first media agent, each of the one or more second media agents to execute a respective controller process for executing respective one or more auxiliary-copy jobs in the plurality of auxiliary-copy jobs, based at least in part on coordination supplied by the coordinator process of the first media agent.

19. The method of claim 11 further comprising:

periodically requesting, by the first media agent, that the storage manager reserve additional data streams for the first auxiliary-copy job.

20. The method of claim 11 further comprising:

reserving, by the storage manager, according to additional requests received from the first media agent while the first auxiliary-copy job is in progress, one or more additional data streams for transferring data in the first auxiliary-copy job.