Fair decentralized throttling in distributed cloud-based systems

• US 10,360,086 B2
• Filed: 08/28/2015
• Issued: 07/23/2019
• Est. Priority Date: 08/28/2015
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Langouev, Ilia, Golosovker, Victor V., Pershin, Aleksey, Wang, Bin
• Primary Examiner(s): Chan, Wing F
• Assistant Examiner(s): Ng, Billy H
• Application Number:
  US14/839,856
• Publication Number:
  US 20170063989A1
• Time in Patent Office: 1,425 Days
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: G06F 9/546 Message passing systems or structures, e.g. queues
• CPC Class Codes: H04L 49/90 Buffering arrangements
• CPC Class Codes: H04L 67/62 Establishing a time schedule for servicing the requests

Abstract

A cloud availability manager configured to execute a recovery workflow that fails over one or more virtual machines (VMs) to and from a cloud computing system. In doing so, the cloud availability manager typically performs multiple operations for each VMs. The operations involve making several application programming interface (API) calls to component APIs of management components within the cloud computing system. To avoid bringing down the entire cloud infrastructure, the cloud availability manager throttles the API calls to other components while executing a recovery workflow. The throttling spans multiple instances (nodes) of the cloud availability manager and involves cooperation from the other management components to ensure the throttling is fair across all tenants of the cloud computing system.

Claims

1. A method of managing a computing instance, comprising:

registering with a central application programming interface (API) gateway a plurality of component APIs of management components within a computing system, wherein the plurality of component APIs includes a first component API;

generating, for each tenant of the computing system, a first queue for synchronous requests to the plurality of component APIs and a second queue for asynchronous requests to the plurality of component APIs;

receiving, at the central API gateway, an API request associated with a first tenant;

when a number of running requests that are associated with the first tenant and are the same type as the API request does not exceed a first limit, forwarding the API request to the first component API; and

when the number of running requests that are associated with the first tenant and are the same type as the API request does exceed the first limit, placing the API request in one of the first queue for synchronous requests and the second queue for asynchronous requests associated with the first tenant based on a state of the first queue and the second queue associated with the first tenant.

2. The method of claim 1, wherein registering the plurality of component APIs of the management components comprises specifying that a request to the component API is one of a synchronous request and an asynchronous request.

3. The method of claim 1, wherein the first queue comprises a first sub-queue for queued tasks for synchronous requests for the first tenant and a second sub-queue for running tasks for synchronous requests for the first tenant, and

wherein the second queue comprises a third sub-queue for queued tasks for asynchronous requests for the first tenant and a fourth sub-queue for running tasks for asynchronous request for the first tenant.

4. The method of claim 1, wherein the request comprises a synchronous request, and wherein placing the API request in one of the first queue for synchronous requests and the second queue for asynchronous requests associated with the first tenant based on the state of the first queue and the second queue associated with the first tenant comprises:

responsive to determining that a number of running synchronous requests associated with the first tenant exceed the first limit associated with the first tenant;

responsive to determining that a number of queued synchronous requests in the first queue associated with the first tenant does not exceeds a second limit associated with the first tenant, placing the request into the first queue for synchronous requests;

responsive to determining that the number of queued synchronous requests in the first queue associated with the first tenant exceeds the second limit associated with the first tenant, rejecting the request from the first tenant.

5. The method of claim 4, further comprising:

subsequent to placing the request into the first queue, forwarding the request to the first component API responsive to determining that the number of running synchronous requests no longer exceeds the first limit associated with the first tenant.

6. The method of claim 1, wherein the request comprises an asynchronous request, and wherein forwarding the API request to the first component API comprises:

responsive to determining that a number of running tasks for asynchronous requests associated with the first tenant does not exceed a third limit associated with the first tenant;

generating a task object for the request, wherein the task object is indicated as running;

transmitting a task identifier of the task object in a response to the first tenant; and

forwarding the request and the task identifier of the task object to the first component API.

7. The method of claim 6, wherein placing the API request in one of the first queue for synchronous requests and the second queue for asynchronous requests associated with the first tenant based on the state of the first queue and the second queue associated with the first tenant further comprises:

responsive to determining that the number of running tasks for asynchronous requests associated with the first tenant exceeds the third limit associated with the first tenant, and responsive to determining that a number of queued tasks for asynchronous requests in the second queue associated with the first tenant does not exceed a fourth limit associated with the first tenant;

generating a task object for the request, wherein the task object is indicated as queued;

transmitting the task identifier in a response to the first tenant; and

placing the request into the second queue for asynchronous requests.

8. The method of claim 6, wherein forwarding the API request to the first component API comprises:

rejecting the request from the first tenant responsive to determining that the number of running tasks for asynchronous requests associated with the first tenant exceeds the third limit associated with the first tenant, and responsive to determining that a number of queued tasks for asynchronous requests in the second queue associated with the first tenant exceeds a fourth limit associated with the first tenant.

9. The method of claim 6, further comprising:

subsequent to placing the request into the second queue, forwarding the request to the first component API responsive to determining that the number of running tasks for asynchronous requests no longer exceeds the third limit associated with the first tenant.

10. The method of claim 6, further comprising:

prior to generating the task object for the request, validating the request with the management component and receiving task type information associated with the request.

11. The method of claim 1, wherein the first limit is shared between the plurality of management components in the computing system.

12. The method of claim 1, wherein the management component comprises a cluster of instances of the management component.

13. A computer system, comprising:

memory configured to store code; and

one or more processors configured to execute the code to;

register with a central application programming interface (API) gateway a plurality of component APIs of management components within a computing system, wherein the plurality of component APIs includes a first component API;

generate, for each tenant of the computing system, a first queue for synchronous requests to the plurality of component APIs and a second queue for asynchronous requests to the plurality of component APIs;

receive, at the central API gateway, an API request associated with a first tenant;

when a number of running requests that are associated with the first tenant and are the same type as the API request does not exceed a first limit, forward the API request to the first component API; and

when the number of running requests that are associated with the first tenant and are the same type as the API request does exceed the first limit, place the API request in one of the first queue for synchronous requests and the second queue for asynchronous requests associated with the first tenant based on a state of the first queue and the second queue associated with the first tenant.

14. The computer system of claim 13, wherein the first queue comprises a first sub-queue for queued tasks for synchronous requests for the first tenant and a second sub-queue for running tasks for synchronous requests for the first tenant, and

wherein the second queue comprises a third sub-queue for queued tasks for asynchronous requests for the first tenant and a fourth sub-queue for running tasks for asynchronous request for the first tenant.

15. The computer system of claim 13, wherein the request comprises a synchronous request, and wherein the one or more processors configured to execute the code to place the API request in one of the first queue for synchronous requests and the second queue for asynchronous requests associated with the first tenant based on the state of the first queue and the second queue associated with the first tenant are further configured to:

responsive to determining that a number of running synchronous requests associated with the first tenant exceed the first limit associated with the first tenant;

responsive to determining that a number of queued synchronous requests in the first queue associated with the first tenant does not exceeds a second limit associated with the first tenant, place the request into the first queue for synchronous requests;

responsive to determining that the number of queued synchronous requests in the first queue associated with the first tenant exceeds the second limit associated with the first tenant, reject the request from the first tenant.

16. The computer system of claim 15, further comprising:

subsequent to placing the request into the first queue, forwarding the request to the first component API responsive to determining that the number of running synchronous requests no longer exceeds the first limit associated with the first tenant.

17. The computer system of claim 13, wherein the request comprises an asynchronous request, and wherein the one or more processors configured to execute the code to forward the API request to the first component API are further configured to:

responsive to determining that a number of running tasks for asynchronous requests associated with the first tenant does not exceed a third limit associated with the first tenant;

generate a task object for the request, wherein the task object is indicated as running;

transmit a task identifier of the task object in a response to the first tenant; and

forward the request and the task identifier of the task object to the first component API.

18. The computer system of claim 17, wherein the one or more processors configured to execute the code to place the API request in one of the first queue for synchronous requests and the second queue for asynchronous requests associated with the first tenant based on a state of the first queue and the second queue associated with the first tenant are further configured to:

responsive to determining that the number of running tasks for asynchronous requests associated with the first tenant exceeds the third limit associated with the first tenant, and responsive to determining that a number of queued tasks for asynchronous requests in the second queue associated with the first tenant does not exceed a fourth limit associated with the first tenant;

generate a task object for the request, wherein the task object is indicated as queued;

transmit the task identifier in a response to the first tenant; and

place the request into the second queue for asynchronous requests.

19. The computer system of claim 17, wherein the one or more processors are configured to execute the code to:

reject the request from the first tenant responsive to determining that the number of running tasks for asynchronous requests associated with the first tenant exceeds the third limit associated with the first tenant, and responsive to determining that a number of queued tasks for asynchronous requests in the second queue associated with the first tenant exceeds a fourth limit associated with the first tenant; and

subsequent to placing the request into the second queue, forward the request to the first component API responsive to determining that the number of running tasks for asynchronous requests no longer exceeds the third limit associated with the first tenant.

20. A non-transitory computer readable medium comprising instructions, which when executed in a computer system, causes the computer system to carry out a method of managing API requests, comprising:

registering with a central application programming interface (API) gateway a plurality of component APIs of management components within a computing system, wherein the plurality of component APIs includes a first component API;

generating, for each tenant of the computing system, a first queue for synchronous requests to the plurality of component APIs and a second queue for asynchronous requests to the plurality of component APIs;

receiving, at the central API gateway, an API request associated with a first tenant;

when a number of running requests that are associated with the first tenant and are the same type as the API request does not exceed a first limit, forwarding the API request to the first component API; and

when the number of running requests that are associated with the first tenant and are the same type as the API request does exceed the first limit, placing the API request in one of the first queue for synchronous requests and the second queue for asynchronous requests associated with the first tenant based on a state of the first queue and the second queue associated with the first tenant.