Adjusting variable limit on concurrent code executions based on communication between frontends

• US 9,977,691 B2
• Filed: 06/29/2016
• Issued: 05/22/2018
• Est. Priority Date: 06/29/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Marriner, Dylan Owen, Roman, Mauricio, Brooker, Marc John, Herwitz, Julian Embry, Reque, Sean
• Primary Examiner(s): Sun, Charlie
• Application Number:
  US15/197,668
• Publication Number:
  US 20180004556A1
• Time in Patent Office: 692 Days
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 9/445 Program loading or initiating bootstrapping G06F9/4401; security arrangements for program loading or initiating G06F21/57
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system

Abstract

Systems and methods are described for adjusting a number of concurrent code executions allowed to be performed for a given user on an on-demand code execution environment or other distributed code execution environments. Such environments utilize pre-initialized virtual machine instances to enable execution of user-specified code in a rapid manner, without delays typically caused by initialization of the virtual machine instances. However, to improve utilization of computing resources, such environments may temporarily restrict the number of concurrent code executions performed on behalf of the given user to a number less than the maximum number of concurrent code executions allowed for the given user. Such environments may adjust the temporary restriction on the number of concurrent code executions based on the number of incoming code execution requests associated with the given user and based on communication among the frontends processing the incoming code execution requests.

Claims

1. A system to provide scaling of computational capacity on an on-demand code execution environment, the system comprising:

a non-transitory data store configured to store one or more program codes associated with a client of the on-demand code execution environment;

one or more hardware computing devices executing a frontend service through specific computer-executable instructions, said frontend service in communication with the non-transitory data store and configured to at least;

receive a request to execute a program code on the on-demand code execution environment, the request including a client ID identifying the client associated with the program code, the client associated with a set of virtual machine instance managers configured to acquire compute capacity on behalf of the client;

calculate a first hash value by applying a hash function on the client ID included in the request;

identify, based on the calculated first hash value, a first virtual machine instance manager on a ring of virtual machine instance managers available on the on-demand code execution environment, the first virtual machine instance manager being in the set of virtual machine instance managers configured to acquire compute capacity on behalf of the client;

request the first virtual machine instance manager to acquire compute capacity for executing the program code on behalf of the client;

receive a throttling response from the first virtual machine instance manager, the throttling response indicating that the first virtual machine instance manager has reached or exceeded a threshold load level;

in response to receiving the throttling response from the first virtual machine instance manager, request current load information associated with the client from one or more other frontend services;

based on the current load information received from the one or more other frontend services, determine that at least one additional virtual machine instance manager should be added to the set of worker managers assigned to the client, the at least one additional virtual machine instance manager including a second virtual machine instance manager; and

execute the program code on behalf of the client on a virtual machine instance acquired by the second virtual machine instance manager.

2. The system of claim 1, wherein the second virtual machine instance manager is associated with a second hash value that is greater than the first hash value, and the second virtual machine instance manager is immediately adjacent to the first virtual machine instance manager on the ring of virtual machine instance managers.

3. The system of claim 1, wherein the frontend service is further configured to:

determine whether a maximum concurrency limit of the client is greater than a current concurrency limit of the client, the maximum concurrency limit of the client defining a range of concurrency limits on a number of concurrent code executions allowed to be performed on the on-demand code execution environment on behalf of the client, and the current concurrency limit of the client indicating a current limit on the number of concurrent code executions allowed to be performed on the on-demand code execution environment on behalf of the client; and

in response to a determination that the maximum concurrency limit of the client is greater than the current concurrency limit of the client, scaling the set of virtual machine instance managers such that a number of virtual machine instance managers in the set is increased.

4. The system of claim 1, wherein the frontend service is further configured to provide an indication to one or more other frontend services that the throttling response was received from the first virtual machine instance manager.

5. A computer-implemented method to provide scaling of computational capacity on an on-demand code execution environment, the computer-implemented method comprising:

receiving a request to execute a program code on the on-demand code execution environment, the request including client information identifying the client associated with the program code, the client associated with a set of virtual machine instance managers configured to acquire compute capacity on behalf of the client;

requesting, based on the client information, a first virtual machine instance manager in the set of virtual machine instance managers to acquire compute capacity for executing the program code on behalf of the client;

receiving a response indicating an inability of the first virtual machine instance manager to acquire compute capacity for executing the program code on behalf of the client;

determining, based on load information associated with one or more frontend modules, whether a number of worker managers in the set should be adjusted, wherein the one or more frontend modules are each configured to cause code executions to be performed in response to receiving corresponding incoming code execution requests;

in response to a determination that the number of worker managers in the set should be adjusted, causing a second virtual machine instance manager different from the first virtual machine instance manager to be added to the set; and

executing at least a portion of the program code on behalf of the client on a virtual machine instance acquired by the second virtual machine instance manager.

6. The computer-implemented method of claim 5, further comprising:

calculating a first hash value by applying a hash function to a client ID of the client; and

identifying the first virtual machine instance manager using the calculated first hash value.

7. The computer-implemented method of claim 5, further comprising providing, to the one or more frontend modules, the indication of the inability of the first virtual machine instance manager to acquire compute capacity.

8. The computer-implemented method of claim 5, further comprising propagating the response indicating the inability of the first virtual machine instance manager to acquire compute capacity to the one or more frontend modules according to a gossip protocol.

9. The computer-implemented method of claim 5, further comprising:

determining, for each frontend module in a group of frontend modules on the on-demand code execution environment, a number of worker managers assigned to the client by frontend module; and

updating, based on the number of worker managers determined for each frontend module in the group, the number of worker managers for one or more frontend modules in the group such that (i) the number of worker managers is decreased for a first frontend module having the highest number of worker managers assigned to the client in the group and (ii) the number of worker managers is increased for a second frontend module having the lowest number of worker managers assigned to the client in the group.

10. The computer-implemented method of claim 5, further comprising:

determining a common scaling strategy based on the load information associated with the one or more frontend modules; and

causing each of the one or more frontend modules to adjust the respective sets of worker managers assigned to the client.

11. The computer-implemented method of claim 5, further comprising determining whether the number of worker managers in the set should be adjusted based on an indication received from a central scaling service in communication with each frontend module on the on-demand code execution environment.

12. The computer-implemented method of claim 5, wherein the one or more frontend modules are each associated with a different iteration order for iterating through the respective sets of worker manager assigned to the client for the frontend module.

13. Non-transitory physical computer storage including computer-executable instructions that, when executed by a computing system, cause the computing system to:

receive a request to execute a program code on an on-demand code execution environment, the request including client information identifying the client associated with the program code, the client associated with a set of virtual machine instance managers configured to acquire compute capacity on behalf of the client;

request, based on the client information, a first virtual machine instance manager in the set of virtual machine instance managers to acquire compute capacity for executing the program code on behalf of the client;

receive a response indicating an inability of the first virtual machine instance manager to acquire compute capacity for executing the program code on behalf of the client;

determine, based on load information associated with one or more frontend modules, whether a number of worker managers in the set should be adjusted, wherein the one or more frontend modules are each configured to cause code executions to be performed in response to receiving corresponding incoming code execution requests;

in response to a determination that the number of worker managers in the set should be adjusted, cause a second virtual machine instance manager different from the first virtual machine instance manager to be added to the set; and

execute at least a portion of the program code on behalf of the client on a virtual machine instance acquired by the second virtual machine instance manager.

14. The non-transitory physical computer storage of claim 13, wherein the instructions further cause the computing system to:

calculate a first hash value by applying a hash function to a client ID of the client; and

identify the first virtual machine instance manager using the calculated first hash value.

15. The non-transitory physical computer storage of claim 13, wherein the instructions further cause the computing system to provide, to the one or more frontend modules, the indication of the inability of the first virtual machine instance manager to acquire compute capacity for executing the program code on behalf of the client.

16. The non-transitory physical computer storage of claim 13, wherein the instructions further cause the computing system to propagate the response indicating the inability of the first virtual machine instance manager to acquire compute capacity to the one or more frontend modules according to a gossip protocol.

17. The non-transitory physical computer storage of claim 13, wherein the instructions further cause the computing system to:

determine, for each frontend module in a group of frontend modules on the on-demand code execution environment, a number of worker managers assigned to the client by frontend module; and

update, based on the number of worker managers determined for each frontend module in the group, the number of worker managers for one or more frontend modules in the group such that (i) the number of worker managers is decreased for a first frontend module having the highest number of worker managers assigned to the client in the group and (ii) the number of worker managers is increased for a second frontend module having the lowest number of worker managers assigned to the client in the group.

18. The non-transitory physical computer storage of claim 13, wherein the instructions further cause the computing system to:

determine a common scaling strategy based on the load information associated with the one or more frontend modules; and

cause each of the one or more frontend modules to adjust the respective sets of worker managers assigned to the client.

19. The non-transitory physical computer storage of claim 13, wherein the instructions further cause the computing system to determine whether the number of worker managers in the set should be adjusted based on an indication received from a central scaling service in communication with each frontend module on the on-demand code execution environment.

20. The non-transitory physical computer storage of claim 13, wherein the one or more frontend modules are each associated with a different iteration order for iterating through the respective sets of worker manager assigned to the client for the frontend module.