Job management and resource allocation

• US 9,575,804 B2
• Filed: 03/27/2015
• Issued: 02/21/2017
• Est. Priority Date: 03/27/2015
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Kochunni, Jaidev Oppath, Liu, Chong, Mishra, Hemant
• Primary Examiner(s): Chen, Qing
• Assistant Examiner(s): THATCHER, CLINT A
• Application Number:
  US14/671,977
• Publication Number:
  US 20160283274A1
• Time in Patent Office: 697 Days
• Field of Search: 718/104
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0622 in relation to access
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5005 to service a request
• CPC Class Codes: G06F 9/5011 the resources being hardware resources other than CPUs, Servers and Terminals
• CPC Class Codes: G06F 9/5016 the resource being the memory
• CPC Class Codes: G06F 9/5022 Mechanisms to release resources

Abstract

A system according to certain aspects improves the process of managing jobs and allocating resources to those jobs. The system can include a job manager that processes and generates resource allocation requests for a plurality of jobs and a resource allocator that allocates computing resources for the plurality of jobs based on the resource allocation requests. The process of allocating resources may involve locking one or more resource objects before allocating the computing resources for the plurality of jobs and unlocking the locked resource objects after the computing resources have been allocated for the plurality of jobs.

Claims

1. A method for managing resource allocation in a data storage system, the method comprising:

with a job manager of a storage management component executing on one or more computing devices in a networked data storage system, the one or more computing devices comprising computer hardware, processing a first data protection job of a plurality of data protection jobs, where performance of the data protection jobs results in application data generated by one or more client computing devices in the data storage system being copied to one or more secondary storage devices to create secondary copies of the application data;

asynchronously requesting allocation of computing resources within the data storage system for use in performing the first data protection job by sending a resource allocation request to a resource allocator of the storage management component, wherein the computing resources include one or more secondary storage devices; and

processing a second data protection job of the plurality of data protection jobs without waiting for the requested allocation of computing resources to be completed by the resource allocator; and

with the resource allocator of the storage management component, in response to the resource allocation request from the job manager, locking one or more resource objects associated with the computing resources within the data storage system such that the one or more resource objects become unmodifiable;

allocating, while the one or more resource objects are locked, a portion of the computing resources for the first data protection job; and

unlocking the one or more resource objects associated with the computing resources such that the one or more resource objects are no longer unmodifiable.

2. The method of claim 1, wherein the job manager is not blocked by the resource allocation request from further processing additional data protection jobs.

3. The method of claim 1, further comprising, with the job manager, after the resource allocation request associated with the first data protection job has been issued sent to the resource allocator but has not yet been fulfilled, performing one or more job management tasks associated with the first data protection job.

4. The method of claim 1 wherein the one or more resource objects comprise tables including metadata associated with the computing resources.

5. The method of claim 1 wherein the job manager and the resource manager are implemented as asynchronous software threads.

6. The method of claim 1, wherein the resource allocation request specifies a first amount of computing resources needed to perform the first data protection job, the method further comprising processing the second data protection before the first amount of computing resources is allocated for the first data protection job based on the resource allocation request.

7. The method of claim 6, further comprising generating, before the first amount of computing resources is allocated for the first data protection job based on the first resource allocation request, another resource allocation request based on the second data protection job, said another resource allocation request specifying a second amount of computing resources needed to perform the second data protection job.

8. The method of claim 1, further comprising determining an amount of computing resources requested for said each data protection job of the plurality of data protection jobs;

acquiring the determined amount of computing resources from an available portion of the computing resources; and

allocating the acquired computing resource to said each data protection job of the plurality of data protection jobs.

9. The method of claim 8, further comprising creating an association between said each data protection job and the respective portion of the computing resources allocated to said each data protection job in one of the locked resource objects.

10. The method of claim 9, wherein said one of the locked resource objects comprises a resource allocation table indicating the allocation of computing resources to the plurality of data protection jobs.

11. The method of claim 1, wherein the one or more resource objects comprise one or more storage devices configured to store data generated during execution of the plurality of data protection jobs.

12. The method of claim 1, further comprising:

receiving, by the resource allocator, a single resource allocation request specifying respective amounts of the computing resources needed for the plurality of data protection jobs; and

processing the single resource allocation request such that the respective amounts are allocated for the plurality of data protection jobs without locking and unlocking the one or more resource objects more than once.

13. The method of claim 1, further comprising:

receiving a plurality of resource allocation requests corresponding to the respective data protection jobs of the plurality of data protection jobs; and

processing the plurality of resource allocation requests in bulk such that respective amounts of the computing resources needed for the plurality of data protection jobs are allocated without locking and unlocking the one or more resource objects more than once.

14. The method of claim 1, wherein the one or more resource objects comprise one or more storage media accessible by the plurality of data protection jobs, wherein the plurality of data protection jobs are configured to store data in the one or more storage media.

15. A system comprising:

a storage manager within a networked data storage system and implemented in one or more hardware processors, the storage manager comprising;

a job manager configured to;

process a first data protection job of a plurality of data protection jobs, where performance of the data protection jobs results in application data generated by one or more client computing devices in the data storage system being copied to one or more secondary storage devices in the data storage system to create secondary copies of the application data;

asynchronously request allocation of computing resources within the data storage system for use in performing the first data protection job by sending a resource allocation request to a resource allocator of the storage manager, wherein the computing resources include one or more secondary storage devices; and

process a second data protection job of the plurality of data protection jobs without waiting for the requested allocation of computing resources to be completed by the resource allocator; and

the resource allocator configured, in response to the resource allocation request from the job manager requests, to;

lock one or more resource objects associated with the computing resources within the data storage system such that the one or more resource objects become unmodifiable;

allocate, while the one or more resource objects are locked, a respective portion of the computing resources for each job of the first plurality of data protection job jobs; and

unlock the one or more resource objects associated with the computing resources such that the one or more resource objects are no longer unmodifiable.

16. The system of claim 15, wherein the job manager is further configured, after the resource allocation request associated with the first data protection job has been issued to the resource allocator but has not yet been fulfilled, to perform one or more job management tasks associated with the first data protection job.

17. The system of claim 15, wherein the resource allocation request specifies a first amount of computing resources needed to perform the first data protection job, the job manager further configured to process the second data protection job before the first amount of computing resources is allocated for the first data protection job based on the first resource allocation request.

18. The system of claim 17, wherein the job manager is further configured to generate, before the first amount of computing resources is allocated for the first data protection job based on the first resource allocation request, another resource allocation request based on the second data protection job, said another resource allocation request specifying a second amount of computing resources needed to perform the second data protection job.

19. The system of claim 15, wherein the resource allocator is further configured to:

receive a single resource allocation request that specifies respective amounts of the computing resources needed for the plurality of data protection jobs; and

process the single resource allocation request such that the respective amounts are allocated for the plurality of data protection jobs without locking and unlocking the one or more resource objects more than once.

20. The system of claim 15, wherein the resource allocator is further configured to:

receive a plurality of resource allocation requests corresponding to the respective data protection jobs of the plurality of data protection jobs; and

process the plurality of resource allocation requests in bulk such that respective amounts of the computing resources needed for the plurality of data protection jobs are allocated without locking and unlocking the one or more resource objects more than once.