Managing contingency capacity of pooled resources in multiple availability zones

• US 9,529,682 B2
• Filed: 12/07/2015
• Issued: 12/27/2016
• Est. Priority Date: 05/15/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: McAlister, Grant Alexander MacDonald, McKelvie, Samuel James, Gupta, Anurag Windlass
• Primary Examiner(s): Mamillapalli, Pavan
• Application Number:
  US14/961,775
• Publication Number:
  US 20160085643A1
• Time in Patent Office: 386 Days
• Field of Search: 707/683
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/815 Virtual
• CPC Class Codes: G06Q 40/04 Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: H04L 41/0668 by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]

Abstract

A network-based services provider may reserve and provision primary resource instance capacity for a given service (e.g., enough compute instances, storage instances, or other virtual resource instances to implement the service) in one or more availability zones, and may designate contingency resource instance capacity for the service in another availability zone (without provisioning or reserving the contingency instances for the exclusive use of the service). For example, the service provider may provision resource instance(s) for a database engine head node in one availability zone and designate resource instance capacity for another database engine head node in another availability zone without instantiating the other database engine head node. While the service operates as expected using the primary resource instance capacity, the contingency resource capacity may be leased to other entities on a spot market. Leases for contingency instance capacity may be revoked when needed for the given service (e.g., during failover).

Claims

1. A computing system, comprising:

a plurality of computing nodes, each of which comprises at least one processor and a memory;

wherein one or more resource instances executing on the plurality of computing nodes implement a storage service, and wherein the storage service comprises a head node and two or more server nodes of a distributed storage service that stores portions of data on one or more storage devices;

wherein one or more other resource instances in a resource instance pool are designated as contingency resource instances for the storage service, but are not reserved for the exclusive use of the storage service;

wherein, while the head node is operating as expected;

the head node is configured to;

receive, from a client of the storage service, one or more write requests, each directed to a respective data item and specifying a modification to be made to the respective data item; and

route information about the specified modifications to particular ones of the server nodes of the distributed storage service;

the server nodes of the distributed storage service are configured to;

apply the specified modifications; and

a resource management component executing on one of the computing nodes is configured to lease one or more of the contingency resource instances to a client, wherein leases for the contingency resource instances are revocable by the resource management component when they are needed for the use of the storage service; and

wherein, in response to determining that the head node is no longer operating as expected, the resource management component is configured to;

revoke a lease for at least one of the one or more contingency resource instances;

reserve the at least one of the contingency resource instances for the use of the storage service; and

instantiate another head node using the at least one of the contingency resource instances.

2. The system of claim 1, wherein the at least one of the contingency resource instances is hosted on a given one of the plurality of computing nodes other than a computing node on which the head node is implemented.

3. The system of claim 1, wherein the contingency resource instances are hosted in a different availability zone or region than one or more availability zones or regions in which the resource instances are hosted.

4. The system of claim 3, wherein a subset of the two or more server nodes of the distributed storage service are implemented using reserved resource instances in the different availability zone or region.

5. The system of claim 1, wherein said determining that the head node is no longer operating as expected comprises detecting a failure of a computing node on which the head node was implemented.

6. The system of claim 1,

wherein, for each of the one or more write requests, said routing comprises:

generating a redo log record representing the modification to be made to the given data item; and

sending the redo log record, but not a data page comprising the given data item, to a particular server node of the distributed storage service that stores a version of the data page comprising the given data item; and

wherein, for each of the one or more write requests, said applying the modification comprises;

receiving the redo log record from the head node;

writing the redo log record to one or more storage devices;

returning, to the head node, an acknowledgement that the redo log record was written; and

subsequent to returning the acknowledgement;

generating a current version of the data page comprising the given data record, wherein to generate the current version of the data page, the particular server node of the distributed storage service is configured to apply the received redo log record and one or more other redo log records representing modifications to the data page to a previously stored version of the data page; and

writing the current version of the data page to one or more storage devices.

7. A method, comprising:

performing by one or more computers;

receiving a request for database services, wherein the request specifies that a database should be deployed on the one or more computers;

reserving primary resource instance capacity for the database, wherein the primary resource instance capacity is sufficient to implement the requested database;

designating contingency resource instance capacity for the database;

while the database operates as expected using the primary resource instance capacity;

receiving and routing database queries to one or more storage nodes using the primary resource instance capacity; and

leasing at least a portion of the contingency resource instance capacity, wherein said leasing comprises providing the at least a portion of the contingency resource instance capacity for a use by an entity other than the database until and unless a trigger condition is detected indicating that it is needed by the database;

detecting a trigger condition indicating that the at least a portion of the contingency resource instance capacity is needed by the database; and

in response to said detecting, revoking a lease on the at least a portion of the contingency resource instance capacity.

8. The method of claim 7, further comprising, in response to said detecting:

reserving the at least a portion of the contingency resource instance capacity for the database; and

instantiating one or more resource instances of the at least a portion of the contingency resource instance capacity as additional resource instances for the database or as replacements for one or more primary resource instances for the database.

9. The method of claim 7,

wherein said reserving comprises reserving the primary resource instance capacity for the database in one or more availability zones;

wherein said designating comprises designating the contingency resource instance capacity for the database in an availability zone other than the one or more availability zones; and

wherein said detecting a trigger condition comprises detecting a failure of a computing node in one of the one or more availability zones on which one or more instances of the primary resource instance capacity were executing.

10. The method of claim 7,

wherein said reserving comprises reserving the primary resource instance capacity for the database in one or more availability zones;

wherein said designating comprises designating the contingency resource instance capacity for the database in an availability zone other than the one or more availability zones; and

wherein said detecting a trigger condition comprises detecting a failure affecting availability of all computing nodes in one of the one or more availability zones.

11. The method of claim 7,

wherein said detecting a trigger condition comprises detecting a failure of a computing node on which one or more resource instances of the primary resource instance capacity were executing;

wherein the method further comprises, in response to said detecting, designating the one or more resource instances of the primary resource instance capacity as contingency resource instances for the database.

12. The method of claim 7, wherein said designating comprises designating one or more particular resource instances as contingency resource instances for the database.

13. The method of claim 7, wherein pricing for designating each of the resource instances of the contingency resource instance capacity is lower than pricing for reserving each of the resource instances of the primary resource instance capacity.

14. The method of claim 7, wherein pricing for leasing each of the resource instances of the contingency resource instance capacity is lower than pricing for reserving each of the resource instances of the primary resource instance capacity.

15. A non-transitory, computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to collectively implement one or more services, wherein the program instructions cause the one or more computers to perform:

provisioning a plurality of resource instances as a pool of resource instances for one of the one or more services;

designating one or more other resource instances as a pool of contingency resource instances for the one of the one or more services;

providing the one of the one or more services using resource instances in the pool of resource instances;

leasing at least a portion of the resource instances in the pool of contingency resource instances for use by another one of the one or more services;

detecting a condition warranting use of the at least a portion of the resource instances in the pool of contingency resource instances by the one of the one or more services;

in response to said detecting;

revoking a lease on the at least a portion of the resource instances in the pool of contingency resource instances;

removing the at least a portion of the resource instances in the pool of contingency resource instances from the pool of contingency resource instances; and

provisioning the at least a portion of the resource instances removed from the pool of contingency resource instances as resource instances for the one of the one or more services.

16. The non-transitory, computer-readable storage medium of claim 15, wherein said detecting a condition comprises detecting a failure condition on the one or more computers.

17. The non-transitory, computer-readable storage medium of claim 15, wherein said detecting a condition comprises detecting that a scaling of the one of the one or more services cannot be supported using the pool of resource instances.

18. The non-transitory, computer-readable storage medium of claim 15, wherein said detecting a condition comprises determining that performance of the one of the one or more services would be improved by provisioning the at least a portion of the resource instances removed from the pool of contingency resource instances as additional resource instances for the one of the one or more services or as replacements for one or more of the resource instances for the one of the one or more services.

19. The non-transitory, computer-readable storage medium of claim 15, wherein the one of the one or more services comprises a database service or a data storage service.