Task-level optimization with compute environments
First Claim
Patent Images
1. A system, comprising:
- a pool of available computing resources of a multi-tenant provider network; and
one or more computing devices having at least one processor and a memory configured to implement a compute environment management system, wherein the compute environment management system is configured to;
receive a job definition defining an array job comprising a first set of one or more tasks and a second set of one or more tasks;
select, a plurality of available computing resources in the pool for execution of the first set of one or more tasks, wherein the plurality of available computing resources comprise a plurality of different hardware configurations;
initiate execution of the first set of one or more tasks using the selected plurality of available computing resources from the pool;
monitor the execution of the first set of one or more tasks by the plurality of different hardware configurations;
generate one or more performance metrics based at least in part on the monitoring of the execution of the first set of one or more tasks;
select one or more recommended hardware configurations from the plurality of different hardware configurations based at least in part on the generated one or more performance metrics;
modify the job definition to indicate the selected one or more recommended hardware configurations;
provision a compute environment based at least in part on the modified job definition, wherein the compute environment comprises a plurality of additional available computing resources from the pool having the indicated one or more recommended hardware configurations; and
initiate execution of the second set of one or more tasks across the compute environment according to the modified job definition such that the second set of one or more of the tasks is executed by the plurality of additional available computing resources having the indicated one or more recommended hardware configurations.
1 Assignment
0 Petitions
Accused Products
Abstract
Methods, systems, and computer-readable media for task-level optimization of compute environments are disclosed. Execution is initiated of one or more tasks using a plurality of computing resources provisioned from a multi-tenant provider network. At least some of the computing resources vary in configuration. One or more metrics are determined that are associated with the execution of the one or more tasks. A configuration of the computing resources is selected based at least in part on the one or more metrics. A modified job definition associated with the one or more tasks is generated. The modified job definition indicates the selected configuration.
60 Citations
15 Claims
-
1. A system, comprising:
-
a pool of available computing resources of a multi-tenant provider network; and one or more computing devices having at least one processor and a memory configured to implement a compute environment management system, wherein the compute environment management system is configured to; receive a job definition defining an array job comprising a first set of one or more tasks and a second set of one or more tasks; select, a plurality of available computing resources in the pool for execution of the first set of one or more tasks, wherein the plurality of available computing resources comprise a plurality of different hardware configurations; initiate execution of the first set of one or more tasks using the selected plurality of available computing resources from the pool; monitor the execution of the first set of one or more tasks by the plurality of different hardware configurations; generate one or more performance metrics based at least in part on the monitoring of the execution of the first set of one or more tasks; select one or more recommended hardware configurations from the plurality of different hardware configurations based at least in part on the generated one or more performance metrics; modify the job definition to indicate the selected one or more recommended hardware configurations; provision a compute environment based at least in part on the modified job definition, wherein the compute environment comprises a plurality of additional available computing resources from the pool having the indicated one or more recommended hardware configurations; and initiate execution of the second set of one or more tasks across the compute environment according to the modified job definition such that the second set of one or more of the tasks is executed by the plurality of additional available computing resources having the indicated one or more recommended hardware configurations. - View Dependent Claims (2, 3, 4)
-
-
5. A computer-implemented method executed by a compute system having at least one processor and a memory, the method comprising:
-
receiving a job definition defining an array job comprising a first set of one or more tasks and a second set of one or more tasks; initiating execution of the first set of one or more tasks on a plurality of different types of available computing resources provisioned from a pool of available computing resources of a multi-tenant provider network, wherein at least some of the different types of available computing resources initiated to execute the first set of one or more tasks vary in configuration; determining one or more metrics that are based on the execution of the first set of one or more tasks by the plurality of different types of available computing resources; selecting a configuration of the varied configurations of the plurality of different types of the available computing resources based at least in part on the determined one or more metrics; generating a modified job definition associated with the second set of one or more tasks, wherein the modified job definition indicates the selected configuration; provisioning, based at least in part on the modified job definition, a compute environment that comprises one or more additional available computing resources from the pool having the indicated configuration; and initiating execution of the second set of one or more tasks across the compute environment according to the modified job definition such that the second set of one or more of the tasks is executed by the one or more additional available computing resources having the indicated configuration. - View Dependent Claims (6, 7, 8, 9, 10, 11, 12)
-
-
13. A non-transitory computer-readable storage medium storing computer-executable program instructions that when executed by one or more processors perform:
-
receiving a job definition defining an array job comprising a first set of one or more tasks and a second set of one or more tasks; initiating execution of the first set of one or more tasks using a plurality of different types of available computing resources provisioned from a pool of available computing resources of a multi-tenant provider network, wherein at least some of the different types of available computing resources initiated to execute the first set of one or more tasks vary in computational or memory characteristics; determining one or more metrics based on the execution of the first set of one or more tasks by the plurality of different types of available computing resources; selecting a configuration of the varied computational or memory characteristics of the plurality of different types of available computing resources based at least in part on the determined one or more metrics; generating a modified job definition associated with the second set of one or more tasks, wherein the modified job definition indicates the selected configuration; provisioning, based at least in part on the modified job definition, a compute environment that comprises one or more additional available computing resources from the pool having the selected configuration indicated by the modified job definition; and initiating execution of the second set of one or more tasks across the compute environment according to the modified job definition such that the second set of one or more tasks is executed by the one or more additional available computing resources having the indicated configuration. - View Dependent Claims (14, 15)
-
Specification