Logical control groups for distributed system resources

• US 10,200,301 B1
• Filed: 03/28/2014
• Issued: 02/05/2019
• Est. Priority Date: 03/28/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Brooker, Marc John, Levy, Marc, Goyal, Tarun, Abrar, Mustafa, Parthasarathy, Madhuvanesh
• Primary Examiner(s): Keehn, Richard G
• Application Number:
  US14/229,319
• Time in Patent Office: 1,775 Days
• CPC Class Codes: H04L 47/78 Architectures of resource allocation
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers

Abstract

A distributed system may implement logical control groups for distributed system resources. A request for a particular system resource may be received at a request routing tier for a distributed system that is implemented by multiple computing systems in different respective availability zones. One or more resource control nodes of a resource control tier for the distributed system that are members of a logical control group for the particular system resource may be discovered. A logical control group for the particular system resource may include at least one resource control node in a different availability zone than the availability zone that includes the particular system resource. The request may then be sent to one of the discovered resource control nodes in the logical control group in order to service the request for the particular system resource.

Claims

1. A system, comprising:

a plurality of computing nodes each included in a respective one of a plurality of different availability zones, wherein each computing node of the plurality of computing nodes is implemented on a respective computing device;

the plurality of computing nodes, configured to implement;

a resource tier, configured to maintain a plurality of system resources;

a resource control tier, comprising a plurality of logical control groups that each include a respective two or more computing nodes of the plurality of computing nodes, wherein for a particular one of the logical control groups, each of the computing nodes of the particular logical control group is configured to service requests for managing a specific system resource of the plurality of system resources corresponding to the respective logical control group, wherein at least one of the respective two or more computing nodes of each of the plurality of logical control groups is included in a different availability zone than an availability zone that includes the corresponding system resource for which the logical control group services requests, and wherein for at least the particular one of the logical control groups at least some of the computing nodes are included in different availability zones with respect to one another;

a request routing tier, configured to;

receive a request for managing a particular system resource of the plurality of system resources;

in response to said request,

discover one or more computing nodes in the respective logical control group for the particular system resource; and

send the request to one of the one or more computing nodes to service the request.

2. The system of claim 1, wherein to discover the one or more computing nodes in the respective logical control group for the particular system resource, the request routing tier is configured to access an entry for the particular system resource in a resource control cache indicating computing nodes configured to service requests for the plurality of system resources.

3. The system of claim 1, wherein to discover the one or more computing nodes in the respective logical control group for the particular system resource, the request routing tier is further configured to:

determine that the entry for the system resource in a resource control cache is empty;

in response to said determining;

select one or more computing nodes of a larger collection of computing nodes implementing the resource control tier that are included in different ones of the plurality of availability zones to query for information describing computing nodes configured to service requests for the particular system resource; and

based, at least in part, on the information received from at least one of the selected one or more computing nodes queried, update the entry in the resource control cache to identify the discovered one or more computing nodes in the respective logical control group for the particular system resource.

4. The system of claim 1, wherein the plurality of computing nodes together implement a network-based service, wherein the plurality of system resources are maintained for one or more clients of the network-based service, and wherein the request is received from a client of the network-based service.

5. A method, comprising:

performing, by a plurality of computing devices included in different respective ones of a plurality of availability zones, wherein the plurality of computing devices together implement a distributed system;

receiving, at a request routing tier providing application layer routing for the distributed system, a request for managing a particular system resource maintained in a resource tier of the distributed system;

in response to receiving said request;

discovering one or more resource control nodes in a logical control group for the particular system resource, the logical control group implemented in a resource control tier of the distributed system, wherein the logical control group comprises a plurality of resource control nodes including the discovered one or more resource control nodes, the plurality of resource control nodes configured to service requests for managing the particular system resource, wherein at least one of the resource control nodes in the logical control group is included in a different availability zone than an availability zone that includes the particular system resource for which the logical control group services requests, and wherein at least some of the resource control nodes in the logical control group are included in different availability zones with respect to one another; and

sending the request received at the request routing tier to one of the resource control nodes to service the request.

6. The method of claim 5, wherein the request is received at the request routing tier from a load balancer.

7. The method of claim 5, wherein the particular system resource is not replicated across one or more of the plurality of availability zones.

8. The method of claim 5, wherein said discovering the one or more resource control nodes in the logical control group implemented in the resource control tier of the distributed system for the particular system resource, comprises:

accessing an entry for the particular system resource in a resource control cache indicating resource control nodes configured to service requests for the resource tier.

9. The method of claim 8, wherein said discovering the one or more resource control nodes in the logical control group implemented in the resource control tier of the distributed system for the particular system resource, further comprises:

determining that the entry for the system resource in the resource control cache is empty;

in response to said determining;

selecting one or more resource control nodes of a larger collection of resource control nodes, including the one or more resource control nodes in the logical control group, of the resource control tier that are included in different ones of the plurality of availability zones, to query for information describing resource control nodes configured to service requests for the particular system resource; and

based, at least in part, on the information received from at least one of the selected one or more resource control nodes queried, updating the entry in the resource control cache to identify the one or more resource control nodes in the logical control group for the particular system resource.

10. The method of claim 8, further comprising:

determining that the one resource control node sent the request is unavailable to service the request;

in response to said determining;

selecting one or more resource control nodes of a larger collection of resource control nodes, including the one or more resource control nodes in the logical control group, of the resource control tier that are included in different ones of the plurality of availability zones to query for information describing resource control nodes configured to service requests for the particular system resource; and

based, at least in part, on the information received from at least one of the selected one or more resource control nodes queried, updating the entry in the resource control cache to identify the one or more resource control nodes in the respective logical control group for the particular system resource.

11. The method of claim 5, wherein the availability zone including the particular system resource is unavailable, and wherein the one resource control node sent the request is included in a different availability zone than the availability zone that includes the particular system resource.

12. The method of claim 11, further comprising recording, at the one resource control node, the request in order to service the request when the availability zone including the particular system resource becomes available.

13. The method of claim 5, further comprising selecting the one resource control node in the logical control group to send the request from among the one or more resource control nodes discovered in the logical control group for the particular system resource.

14. The method of claim 5, further comprising performing, among a larger collection of resource control nodes, including the one or more resource control nodes in the logical control group, a gossip-based synchronization protocol to synchronize respective metadata describing other resource control nodes maintained at each resource control node of the larger collection of resource control nodes.

15. The method of claim 5, wherein the plurality of computing devices together implement the distributed system as a network-based service, wherein the particular system resource is one of a plurality of system resources that are maintained for one or more clients of the network-based service, and wherein the request is received from a client of the network-based service associated with the particular system resource.

16. A non-transitory, computer-readable storage medium, storing program instructions that when executed by a plurality of computing devices that together implement a distributed system cause the one or more computing devices to implement:

receiving, at a request routing tier of the distributed system, a request for a particular system resource maintained in a resource tier of the distributed system, wherein the particular system resource is one of a plurality of different system resources in the resource tier that are each included in a respective availability zone of a plurality of availability zones;

in response to receiving said request;

discovering one or more resource control nodes in a logical control group for the particular system resource, the logical control group implemented in a resource control tier of the distributed system, wherein the logical control group comprises a plurality of resource control nodes, including the discovered one or more resource control nodes, the plurality of resource control nodes configured to service requests for managing the particular system resource, wherein at least one of the resource control nodes in the logical control group is included in a different availability zone than the respective availability zone that includes the particular system resource for which the logical control group services requests, and wherein at least some of the resource control nodes in the logical control group are included in different availability zones with respect to one another; and

sending the request received at the request routing tier to one of the one or more resource control nodes to service the request.

17. The non-transitory, computer-readable storage medium of claim 16, wherein each of the plurality of availability zones provides a respective failure scope such that respective portions of the request routing tier, the resource control tier, and the resource tier included in an availability zone operate irrespective of failures in another availability zone.

18. The non-transitory, computer-readable storage medium of claim 16, wherein in said discovering the one or more resource control nodes in the logical control group for the particular system resource implemented in the resource control tier of the distributed system, the program instructions cause the plurality of computing devices to implement:

accessing an entry for the particular system resource in a resource control cache, the entry indicating resource control nodes configured to service requests for the particular system resource.

19. The non-transitory, computer-readable storage medium of claim 18, wherein in said discovering the one or more resource control nodes in the logical control group for the particular system resource, the program instructions further cause the plurality of computing devices to implement:

determining that the entry for the particular system resource in the resource control cache is empty;

in response to said determining;

selecting one or more resource control nodes of a larger collection of resource control nodes of the resource control tier, including the one or more resource control nodes in the logical control group implemented in different ones of the plurality of availability zones to query for information describing resource control nodes configured to service requests for the particular system resource; and

based, at least in part, on the information received from at least one of the selected one or more resource control nodes queried, updating the entry in the resource control cache to identify the one or more resource control nodes in the respective logical control group for the particular system resource.

20. The non-transitory, computer-readable storage medium of claim 16, wherein the respective availability zone including the particular system resource is available, wherein the one resource control node sent the request is implemented in a different availability zone than the respective availability zone that includes the particular system resource, and wherein the program instructions further cause the plurality of computing devices to implement:

performing the service request at the one resource control node with respect to the particular system resource in the respective availability zone.

21. The non-transitory, computer-readable storage medium of claim 16, wherein the program instructions further cause the plurality of computing devices to implement selecting the one resource control node in the logical control group to send the request from among the one or more resource control nodes discovered for the particular system resource.

22. The non-transitory, computer-readable storage medium of claim 16, wherein the program instructions further cause the plurality of computing devices to implement performing, among a larger collection of resource control nodes, including the one or more resource control nodes in the logical control group, that implement the resource control tier, a gossip-based synchronization protocol to synchronize respective metadata describing other resource control nodes maintained at each resource control node of the larger collection of resource control nodes.

23. The non-transitory, computer-readable storage medium of claim 16, wherein the plurality of computing nodes together implement a network-based, block-based storage service, wherein the particular system resource is a one of a plurality of data volumes that are maintained for one or more clients of the network-based, block-based storage service, and wherein the request is received from a client of the network-based, block-based storage service associated with the data volume.