SYSTEM AND METHOD FOR PROVIDING INTELLIGENT PRE-STAGING OF DATA IN A COMPUTE ENVIRONMENT

• US 20090144215A1
• Filed: 12/29/2008
• Published: 06/04/2009
• Est. Priority Date: 03/13/2004
• Status: Active Grant

• Current Assignee: III Holdings 12, LLC (Intellectual Ventures LLC)
• Original Assignee: Cluster Resources, Inc. (Ala Services LLC)
• Inventors: Jackson, David B.
• Granted Patent: US 8,763,000 B2
• Time in Patent Office: Days
• US Class Current: 706/46
• CPC Class Codes: G06F 15/161 Computing infrastructure, e.g. computer clusters, blade chassis or hardware partitioning casings, cabinets, racks or drawers for data centers H05K5/00
• CPC Class Codes: G06F 2209/5014 Reservation
• CPC Class Codes: G06F 2209/5015 Service provider selection
• CPC Class Codes: G06F 2209/5022 Workload threshold
• CPC Class Codes: G06F 2209/503 Resource availability
• CPC Class Codes: G06F 2209/506 Constraint
• CPC Class Codes: G06F 2209/508 Monitor
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5011 the resources being hardware resources other than CPUs, Servers and Terminals
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: G06F 9/505 considering the load
• CPC Class Codes: G06F 9/5072 Grid computing

Abstract

A system, computer-readable medium and method for performing intelligent data pre-staging for a job submitted to a cluster environment. The method aspect comprises determining availability of compute resources including availability timeframes to process the submitted job, determining data requirements for processing the job and determining a co-allocation in time reservation.

Claims

1. A computer-implemented method of pre-staging data for a job submitted to a compute environment, the method comprising:

determining availability of compute resources in a compute environment including availability timeframes to process the submitted job;

determining data requirements for processing the submitted job;

based on the determined availability and determined data requirements, establishing a data stage-in reservation that starts at a first start time prior to a second start time for a compute reservation for the submitted job; and

performing data pre-staging according to the data pre-staging reservation at the first start time and thereafter processing the job according to the compute reservation at the second start time.

2. The computer-implemented method of claim 1, wherein establishing the data stage-in reservation is based at least in part on (1) creating an availability range list from a calculation of existing resource guarantees and reservations in place and (2) converting the availability range list into a start range list.

3. The computer-implemented method of claim 1, wherein establishing the data stage-in reservation, further comprising:

(1) requesting resources for a first step in the job based on at least one of the determined availability and the determined data requirements;

(2) calculating existing resource guarantees and reservations already in place to create an availability range list;

(3) converting the availability range list into a start range list;

(4) requesting another resource for a next step in the job process and returning to step

(2) until all resources for all steps in the job are requested;

(5) shifting start ranges in the start range list by an offset and performing an intersection operation on a combination start range;

(6) shifting the start ranges back by a negative of the offset, wherein resulting information provides when to start each potential data stage-in reservation; and

(7) establishing the data stage-in reservation based on the resulting information that processes when to start each potential data stage-in reservation.

4. The computer-implemented method of claim 3, further comprising:

(1) presenting to a user a final list of selectable starting times for a data stage-in reservation; and

(2) upon receiving a user selection of a data stage-in reservation start time, establishing a data stage-in reservation.

5. The computer-implemented method of claim 1, wherein compute resources are available prior to completion of data pre-staging.

6. The computer-implemented method of claim 1, wherein establishing the data stage-in reservation is based at least in part on an analysis of whether each step in a job can be processed based on existing resource guarantees and reservations.

7. A system that pre-stages data for a job submitted to a compute environment, the system comprising:

a processor;

a module configured to control the processor to determine availability of compute resources in a compute environment including availability timeframes to process a submitted job;

a module configured to control the processor to determine data requirements for processing the submitted job;

a module configured to control the processor, based on the determined availability and determined data requirements, to establish a data stage-in reservation that starts at a first start time prior to a second start time for a compute reservation for the submitted job; and

a module configured to control the processor to perform data pre-staging according to the data pre-staging reservation at the first start time and thereafter processing the job according to the compute reservation at the second start time.

8. The system of claim 7, wherein the module configured to control the processor to further establishes the data stage-in reservation based at least in part on (1) creating an availability range list from a calculation of existing resource guarantees and reservations in place and (2) converting the availability range list into a start range list.

9. The system of claim 7, wherein a module configured to control the processor to establish the data stage-in reservation, further establishes the data stage-in reservation by:

(1) requesting resources for a first step in the job based on at least one of the determined availability and the determined data requirements;

(2) calculating existing resource guarantees and reservations already in place to create an availability range list;

(3) converting the availability range list into a start range list;

(4) requesting another resource for a next step in the job process and returning to step (2) until all resources for all steps in the job are requested;

(5) shifting start ranges in the start range list by an offset and performing an intersection operation on a combination start range;

(6) shifting the start ranges back by a negative of the offset, wherein resulting information provides when to start each potential data stage-in reservation; and

(7) establishing the data stage-in reservation based on the resulting information that processes when to start each potential data stage-in reservation.

10. The system of claim 9, further comprising a module configured to control the processor to present to a user a final list of selectable starting times for a data stage-in reservation and a module configured, to control the process, upon receiving a user selection of a data stage-in reservation start time, to establish a data stage-in reservation.

11. The system of claim 7, wherein compute resources are available prior to completion of data pre-staging.

12. The system of claim 7, wherein the module configured to control the processor to establish the data stage-in reservation further establishes the data stage in reservation based on at least in part an analysis of whether each step in a job can be processed based on existing resource guarantees and reservations.

13. A tangible computer-readable medium storing instructions for controlling a computing device to pre-stage data for a job submitted to a compute environment, the instructions comprising:

determining availability of compute resources in a compute environment including availability timeframes to process the submitted job;

determining data requirements for processing the submitted job;

based on the determined availability and determined data requirements, establishing a data stage-in reservation that starts at a first start time prior to a second start time for a compute reservation for the submitted job; and

performing data pre-staging according to the data pre-staging reservation at the first start time and thereafter processing the job according to the compute reservation at the second start time.

14. The tangible computer readable medium of claim 13, wherein establishing the data stage-in reservation is based at least in part on (1) creating an availability range list from a calculation of existing resource guarantees and reservations in place and (2) converting the availability range list into a start range list.

15. The tangible computer readable medium of claim 13, wherein establishing the data stage-in reservation, further comprising:

(1) requesting resources for a first step in the job based on at least one of the determined availability and the determined data requirements;

(2) calculating existing resource guarantees and reservations already in place to create an availability range list;

(3) converting the availability range list into a start range list;

(4) requesting another resource for a next step in the job process and returning to step (2) until all resources for all steps in the job are requested;

(5) shifting start ranges in the start range list by an offset and performing an intersection operation on a combination start range;

(6) shifting the start ranges back by a negative of the offset, wherein resulting information provides when to start each potential data stage-in reservation; and

(7) establishing the data stage-in reservation based on the resulting information that processes when to start each potential data stage-in reservation.

16. The tangible computer readable medium of claim 15, the instructions further comprising:

(1) presenting to a user a final list of selectable starting times for a data stage-in reservation; and

(2) upon receiving a user selection of a data stage-in reservation start time, establishing a data stage-in reservation.

17. The tangible computer readable medium of claim 13, wherein compute resources are available prior to completion of data pre-staging.

18. The tangible computer readable medium of claim 13, wherein establishing the data stage-in reservation is based at least in part on an analysis of whether each step in a job can be processed based on existing resource guarantees and reservations.