Assigning data agent proxies for executing virtual-machine secondary copy operations including streaming backup jobs

• US 10,298,710 B2
• Filed: 04/05/2018
• Issued: 05/21/2019
• Est. Priority Date: 06/19/2015
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Dornemann, Henry Wallace
• Primary Examiner(s): Dailey, Thomas J
• Application Number:
  US15/946,116
• Publication Number:
  US 20180295201A1
• Time in Patent Office: 411 Days
• Field of Search: None
• CPC Class Codes: H04L 41/042 comprising distributed management centres cooperatively managing the network
• CPC Class Codes: H04L 41/0856 by backing up or archiving configuration information
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54

Abstract

A comprehensive approach to streaming backups for virtual machines (“VMs”) in a storage management system comprises improvements to the assignment of data agent proxies for VM secondary copy operations. New considerations in performing a VM streaming backup job include without limitation: determining and enforcing a system-wide per-proxy limit of concurrent data streams; generating an ordered priority list of the VMs to be backed up as a basis for choosing which proxies will back up the respective VM, though the illustrative system may not strictly adhere to the priority list based on further considerations; identifying a next available proxy based on data stream utilization at the proxy; and dynamically re-generating the priority list and re-evaluating considerations if some VMs become “stranded” due to a failure to be backed up. Secondary copy operations are distributed to proxies in ways that improve the chances of successfully completing VM streaming backups.

Claims

1. A method comprising:

designating, by a storage manager in a storage management system, a first data agent to coordinate a first backup job for performing secondary copy operations for a first set of virtual machines in the storage management system,

wherein the first data agent executes on a first virtual machine, and

wherein the first virtual machine executes on a first computing device comprising one or more processors and non-transitory random access memory;

designating, by the storage manager, a second data agent to act as controller in the first backup job coordinated by the first data agent,

wherein the second data agent executes on a second computing device comprising one or more processors and non-transitory random access memory;

determining, by the first data agent, a respective maximum limit of concurrent data streams assignable by the first data agent to each of the first virtual machine and the second computing device for performing one or more backup jobs in the storage management system, including the first backup job;

wherein a maximum limit of concurrent data streams assignable to the first virtual machine is based on a processing capacity and an amount of random access memory of the first virtual machine; and

wherein a maximum limit of concurrent data streams assignable to the second computing device is based on a processing capacity and an amount of random access memory of the second computing device.

2. The method of claim 1 further comprising:

assigning, by the first data agent coordinating the first backup job, a first data stream to the first virtual machine, wherein the first data stream originates at a first storage device comprising virtual-machine data being backed up in the first backup job and terminates at the first data agent that executes on the first virtual machine; and

wherein the first virtual machine concurrently uses no more than the maximum limit of concurrent data streams assignable by the first data agent to the first virtual machine.

3. The method of claim 1, wherein the first data agent and the second data agent are virtual-server data agents configured for backup of virtual machines in the storage management system.

4. The method of claim 1, wherein the first set of virtual machines comprises a first subset of virtual machines that execute on a first host computing device, and a second subset of virtual machines that execute on a second host computing device.

5. The method of claim 1, wherein the first set of virtual machines comprises a first subset of virtual machines that execute on the first computing device on which the first virtual machine also executes, and further comprises a second subset of virtual machines that execute on another computing device distinct from the first computing device and the second computing device.

6. The method of claim 1, wherein the first virtual machine and at least some of the first set of virtual machines execute on the first computing device.

7. The method of claim 1, wherein the first virtual machine and one or more of the first set of virtual machines execute on the first computing device, and wherein other one or more of the first set of virtual machines execute on another computing device that is distinct from the first computing device and from the second computing device.

8. The method of claim 1, wherein the first data agent coordinates the first backup job for the first set of virtual machines, and wherein the first virtual machine and one or more of the first set of virtual machines execute on the first computing device, and wherein other one or more of the first set of virtual machines execute on another computing device that is distinct from the first computing device and from the second computing device.

9. The method of claim 1 further comprising:

collecting, by the first data agent, job-related information for executing the first backup job; and

wherein the job-related information comprises one or more of;

identities of the first set of virtual machines to be backed up, one or more operational properties of each virtual machine in the first set, information about one or more storage devices storing virtual-machine data to be backed up, and a maximum number of concurrent data streams allowed to be used concurrently while performing the first backup job.

10. The method of claim 1, wherein the maximum limit of concurrent data streams assignable to the second computing device is further based on a number of processors in the second computing device.

11. A storage management system comprising:

a storage manager comprising one or more processors and non-transitory random access memory;

a first computing device comprising one or more processors and non-transitory random access memory;

a second computing device comprising one or more processors and non-transitory random access memory;

wherein the storage manager is configured to designate a first data agent to coordinate a first backup job for performing secondary copy operations for a first set of virtual machines in the storage management system, wherein the first data agent executes on a first virtual machine hosted by the first computing device;

wherein the storage manager is further configured to designate a second data agent to act as controller in the first backup job coordinated by the first data agent, wherein the second data agent executes on the second computing device;

wherein the first data agent is configured to determine a respective maximum limit of concurrent data streams assignable by the first data agent to each of the first virtual machine and the second computing device for performing the first backup job;

wherein a maximum limit of concurrent data streams assignable to the first virtual machine is based on a processing capacity and an amount of random access memory of the first virtual machine; and

wherein a maximum limit of concurrent data streams assignable to the second computing device is based on a processing capacity and an amount of random access memory of the second computing device.

12. The system of claim 11 wherein the first data agent coordinating the first backup job is further configured to assign a first data stream to the first virtual machine, wherein the first data stream originates at a first storage device comprising virtual-machine data being backed up in the first backup job and terminates at the first data agent that executes on the first virtual machine;

and

wherein the first virtual machine concurrently uses no more than the maximum limit of concurrent data streams assignable by the first data agent to the first virtual machine.

13. The system of claim 11 wherein the second computing device hosts a second virtual machine on which the second data agent executes.

14. The system of claim 11, wherein the first virtual machine and at least some of the first set of virtual machines execute on the first computing device.

15. The system of claim 11, wherein the first data agent coordinates the first backup job for the first set of virtual machines, and wherein the first virtual machine and one or more of the first set of virtual machines execute on the first computing device, and wherein other one or more of the first set of virtual machines execute on another computing device that is distinct from the first computing device and from the second computing device.

16. A storage management system comprising:

a first computing device configured to;

execute a coordinator data agent that communicates with one or more controller data agents in regard to a backup job to be performed for virtual machines in the storage management system, wherein each controller data agent executes on a respective second computing device distinct from the first computing device,

collect information about the backup job to be performed,

further collect information about operational properties of the first computing device and each respective second computing device, which are candidates for executing at least part of the backup job to be performed, wherein for a given candidate computing device the operational properties include a processing capacity and an amount of random access memory of the given candidate computing device, and

determine a respective maximum limit of concurrent data streams assignable by the coordinator data agent to each candidate computing device for transferring data in the backup job to be performed, wherein for a given candidate computing device the respective maximum limit of concurrent data streams is based on the operational properties collected about the given candidate computing device; and

wherein any given computing device cannot concurrently use more than the respective maximum limit of concurrent data streams in the course of performing the backup job in the storage management system.

17. The system of claim 16, wherein the information about a backup job comprises one or more of:

identities of the virtual machines to be backed up in the backup job, operational properties of the virtual machines to be backed up in the backup job, and a number of concurrent data streams allowed for performing the backup job.

18. The system of claim 16, wherein the first computing device is implemented as a virtual machine executing on a host computing device comprising one or more processors and random access memory.

19. The system of claim 16, wherein the first computing device is implemented as a virtual machine executing on a host computing device that comprises one or more processors and random access memory and also hosts some of the virtual machines being backed up in the backup job.