Message-based computation request scheduling

• US 10,048,974 B1
• Filed: 09/30/2014
• Issued: 08/14/2018
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Wagner, Timothy Allen, Manwarin, Derek Steven, Reque, Sean Philip, Thomas, Dylan Chandler
• Primary Examiner(s): Dao, Thuy
• Application Number:
  US14/502,589
• Time in Patent Office: 1,414 Days
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/50 Allocation of resources, e.g. of the central processing unit [CPU]
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals

Abstract

A system for routing requests to execute user code based on how frequently the user code is executed is provided. The system may be configured to receive a request to execute user code on a virtual compute system, where the virtual compute system comprises multiple fleets of virtual machine instances. The system may be further configured to determine whether the user code associated with the request satisfies one or more usage criteria, and based on the determination, route the request to the appropriate fleet of virtual machine instances.

Claims

1. A computer-implemented method comprising:

as implemented by one or more computing devices configured with specific executable instructions,

receiving a request to execute a first program code on a virtual compute system, the request including a first identifier identifying the first program code requested to be executed on the virtual compute system, wherein the virtual compute system comprises at least a first fleet of virtual machine instances configured to process low volume requests and a second fleet of virtual machine instances configured to process high volume requests;

accessing, based on the first identifier included in the request, first program code data associated with the first program code from an electronic data store, wherein the first program code data indicates first execution frequency of the first program code, wherein the electronic data store is configured to store at least program code data associated with a plurality of program codes, the program code data indicating frequency of execution of the plurality of program codes on the virtual compute system; and

based on the first execution frequency indicated by the first program code data, routing the request to the first fleet of virtual machine instances configured to process low volume requests.

2. The computer-implemented method of claim 1, further comprising:

receiving a request to execute a second program code on the virtual compute system;

accessing, from the electronic data store, second program code data associated with the second program code, wherein the second program code data indicating second execution frequency of the second program code; and

based on the second execution frequency, causing the second program code to be executed on a virtual machine instance selected from the second fleet of virtual machine instances configured to process high volume requests.

3. The computer-implemented method of claim 1, further comprising:

recording, for a program code executed on the virtual compute system, program code data associated with the program code, wherein the program code data at least includes a client ID associated with the program code and a usage status associated with the program code.

4. The computer-implemented method of claim 1, further comprising:

receiving an instruction to adjust a capacity of the first fleet of virtual machine instances; and

adjusting a number of worker managers configured to manage the virtual machine instances of the first fleet based on the received instruction.

5. The computer-implemented method of claim 1, wherein the first execution frequency comprises one or more of (i) a number of times the first program code has been executed on the virtual compute system, (ii) a transactions-per-second value associated with the first program code, and (iii) an indication of whether the first program code is a high-frequency code or a low-frequency code.

6. A system for providing low-latency computational capacity from a virtual compute fleet, the system comprising:

an electronic data store configured to store at least usage data associated with one or more program codes executed on the virtual compute fleet,

wherein the virtual compute fleet comprises a low volume fleet of virtual machine instances configured to process low volume requests and a high volume fleet of virtual machine instances configured to process high volume requests, wherein the low volume fleet contains a group of worker managers, each worker manager configured to manage one or more virtual machine instances of the low volume fleet and has one or more hash values associated therewith; and

one or more hardware computing devices executing a frontend service through specific computer-executable instructions, said frontend service in communication with the electronic data store, and configured to at least;

receive a request to execute a first program code on the virtual compute fleet;

access, from the electronic data store, program code data associated with the first program code to be executed, wherein the program code data is indicative of a first frequency at which the first program code has been executed on the virtual compute fleet;

determine that the first frequency at which the first program code has been executed on the virtual compute fleet does not exceed a threshold frequency;

calculate a specific hash value based on at least a portion of information included in the request;

determine a first worker manager of the group of worker managers of the low volume fleet corresponding to the calculated specific hash value; and

route the request to the first worker manager corresponding to the calculated specific hash value.

7. The system of claim 6, wherein the frontend service is further configured to:

determine that a request to execute a second program code on the virtual compute fleet satisfies one or more usage criteria; and

route the request to execute the second program code on the virtual compute fleet to a second worker manager of the high volume fleet that is associated with the frontend service, wherein the second worker manager is configured to manage the virtual machine instances of the high volume fleet.

8. The system of claim 6, further comprising a usage data manager configured to record, for each program code executed on the virtual compute fleet, usage data associated with the program code, wherein the usage data at least includes a client ID associated with the program code and a usage status associated with the program code, wherein the usage data manager is configured to determine the usage status based an execution frequency of the program code.

9. A system, comprising:

an electronic data store configured to store at least program code data associated with a plurality of program codes, the program code data indicating frequency of execution of the plurality of program codes on a virtual compute system, wherein the virtual compute system comprises at least a first fleet of virtual machine instances configured to process low volume requests and a second fleet of virtual machine instances configured to process high volume requests; and

a frontend service comprising one or more hardware computing devices executing specific computer-executable instructions that, when executed, causes the one or more hardware computing devices to at least;

receive a request to execute a first program code on a virtual compute system, the request including a first identifier identifying the first program code requested to be executed on the virtual compute system;

access, based on the first identifier included in the request, first program code data associated with the first program code from the electronic data store, wherein the first program code data indicates first execution frequency of the first program code; and

based on the first execution frequency indicated by the first program code data, route the request to the first fleet of virtual machine instances configured to process low volume requests.

10. The system of claim 9, wherein the frontend service is further configured to:

receive a request to execute a second program code on the virtual compute system;

access, from the electronic data store, second program code data associated with the second program code, wherein the second program code data indicating second execution frequency of the second program code; and

based on the second execution frequency, cause the second program code to be executed on a virtual machine instance selected from the second fleet of virtual machine instances configured to process high volume requests.

11. The system of claim 9, further comprising a usage data manager configured to record, for a program code executed on the virtual compute system, program code data associated with the program code, wherein the program code data at least includes a client ID associated with the program code and a usage status associated with the program code, wherein the usage data manager is further configured to determine the usage status based on how frequently the program code has been executed on the virtual compute system.

12. The system of claim 9, wherein the frontend service is further configured to:

receive an instruction to adjust a capacity of the first fleet of virtual machine instances; and

adjust a number of worker managers configured to manage the virtual machine instances of the first fleet based on the received instruction.

13. The system of claim 9, wherein the first execution frequency comprises one or more of (i) a number of times the first program code has been executed on the virtual compute system, (ii) a transactions-per-second value associated with the first program code, and (iii) an indication of whether the first program code is a high-frequency code or a low-frequency code.

14. The system of claim 9, wherein the frontend service is further configured to, in response to routing the request to the first fleet, determine a virtual machine instance for executing the first program code of the user, at least in part by:

calculating a hash value associated with the request; and

forwarding the request to a worker manager associated with the calculated hash value, wherein the worker manager is configured to manage at least a portion of the first fleet of virtual machine instances.

15. The system of claim 9, further comprising:

one or more additional frontend services each associated with a different fleet of virtual machine instances, wherein the frontend service and said one or more additional frontend services are implemented on separate hardware computing devices located in different logical failure zones implemented at least in different physical data centers, different power grids, different physical racks, different networks, different subnets, different geographic regions, different security boundaries, or different availability zones; and

a load balancer configured to distribute requests to execute program codes on the virtual compute system based on an individual capacity of the frontend services.

16. A computer-readable, non-transitory storage medium storing computer executable instructions that, when executed by one or more computing devices, configure the one or more computing devices to perform operations comprising:

receiving a request to execute a first program code on a virtual compute system, the request including a first identifier identifying the first program code requested to be executed on the virtual compute system, wherein the virtual compute system comprises at least a first fleet of virtual machine instances configured to process low volume requests and a second fleet of virtual machine instances configured to process high volume requests;

accessing, based on the first identifier included in the request, first program code data associated with the first program code from an electronic data store, wherein the first program code data indicates first execution frequency of the first program code, wherein the electronic data store is configured to store at least program code data associated with a plurality of program codes, the program code data indicating frequency of execution of the plurality of program codes on the virtual compute system; and

based on the first execution frequency indicated by the first program code data, routing the request to the first fleet of virtual machine instances configured to process low volume requests.

17. The computer-readable, non-transitory storage medium of claim 16, wherein the operations further comprise:

receiving a request to execute a second program code on the virtual compute system;

accessing, from the electronic data store, second program code data associated with the second program code, wherein the second program code data indicating second execution frequency of the second program code; and

causing the second program code to be executed on a virtual machine instance selected from the second fleet of virtual machine instances configured to process high volume requests.

18. The computer-readable, non-transitory storage medium of claim 16, wherein the operations further comprise recording, for a program code executed on the virtual compute system, program code data associated with the program code, wherein the program code data at least includes a client ID associated with the program code and a usage status associated with the program code.

19. The computer-readable, non-transitory storage medium of claim 16, wherein the operations further comprise:

receiving an instruction to adjust a capacity of the first fleet of virtual machine instances; and

adjusting a number of worker managers configured to manage the virtual machine instances of the first fleet based on the received instruction.

20. The computer-readable, non-transitory storage medium of claim 16, wherein the first execution frequency comprises one or more of (i) a number of times the first program code has been executed on the virtual compute system, (ii) a transactions-per-second value associated with the first program code, and (iii) an indication of whether the first program code is a high-frequency code or a low-frequency code.