Managing test services in a distributed production service environment

• US 10,521,258 B2
• Filed: 03/03/2017
• Issued: 12/31/2019
• Est. Priority Date: 09/29/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Burruss, Brian Todd, Guthrie, II, John Luther, Olson, Marc Stephen, Parthasarathy, Madhuvanesh
• Primary Examiner(s): Dorais, Craig C
• Application Number:
  US15/449,801
• Publication Number:
  US 20180165112A1
• Time in Patent Office: 1,033 Days
• Field of Search: None
• CPC Class Codes: G06F 2009/45579 I/O management, e.g. providing access to device drivers or storage
• CPC Class Codes: G06F 2009/45583 Memory management, e.g. access or allocation
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Systems and methods are described for a storage processing service that processes multiple storage commands. The storage processing service uses tags from test storage commands to determine whether a test storage service is to be instantiated that reflects a corresponding production service. Test storage commands with the same tag are tested on that test service. Additionally, the storage processing service determines a strategy for testing processes on production services when the storage system is overloaded. In one embodiment, the test service manager can determine to stop testing processes for a period of time, and issue a shed command that queues or sheds test storage commands. Advantageously, a shed command, while active at a storage processing service, may alleviate the overload on production services. The test service manager can continue to monitor the storage system to determine whether the overload continues to exist.

Claims

1. A system comprising:

a storage processing service comprising a production storage volume and a test storage volume, the storage processing service executed on one or more storage computing devices and configured to at least;

obtain a first storage request and a second storage request from at least one of a plurality of virtual computing instances, the first storage request being for the production storage volume and the second storage request being for the test storage volume, the first storage request including a first tag indicating that the first storage request is a production storage request and the second storage request including a second tag indicating that the second storage request is a test storage request;

route the first storage request for processing based at least in part on the first tag; and

route the second storage request for processing based at least in part on the second tag; and

a test service manager executed on one or more computing devices and configured to at least;

identify, based at least in part on a first result of processing the first storage request and a second result of processing the second storage request, a storage traffic overload on the storage processing service; and

determine, based at least in part on the storage traffic overload, that the storage processing service is to stop processing test storage requests.

2. The system of claim 1, wherein the storage processing service is further configured to collect monitoring information based at least in part on the first result of processing the first storage request and the second result of processing the second request.

3. The system of claim 2, wherein the storage processing service is further configured to at least:

process the first storage request to generate the first result; and

process the second storage request to generate the second result.

4. The system of claim 2, wherein the test service manager is further configured to at least:

analyze the monitoring information to identify the storage traffic overload on the storage processing service; and

determine, based at least in part on the monitoring information, that the storage processing service is to stop processing test storage requests.

5. The system of claim 1, wherein the storage processing service is further configured to at least:

obtain a third storage request from at least one of the plurality of virtual computing instances, the third storage request being for the production storage volume and including a third tag indicating that the third storage request is an additional production storage request;

route the third storage request for processing based at least in part on the third tag; and

process the third storage request to generate a third result.

6. A computer-implemented method for processing test storage commands on a storage processing service, the storage processing service comprising a test service and a first production service in a production computing environment, the computer-implemented method comprising:

obtaining, at the first production service, a first storage command including a first tag indicating that the first storage command is a production storage command and a second storage command, including a second tag indicating that the second storage command is a test storage command;

routing, based at least in part on the second tag, the production storage command to the first production service to generate a production storage processing result;

routing, based at least in part on the second tag, the test storage command to the test service to generate a test storage processing result, wherein the test service is a modification to the first production service, and wherein the test storage command is implemented, at least in part, by a resource of the production computing environment;

collecting monitoring information based at least in part on the production storage processing result and the test storage processing result; and

modifying processing of a subsequent test storage command by the storage processing service at least in part based on the collected monitoring information.

7. The computer-implemented method of claim 6 further comprising analyzing the monitoring information to identify a storage traffic overload on the storage processing service, the storage traffic overload corresponding to an overload of production storage commands on a threshold level of storage volumes, wherein modifying processing of a subsequent test storage command comprises determining that the storage processing service is to stop processing test storage commands.

8. The computer-implemented method of claim 7, wherein analyzing the monitoring information comprises determining that the test storage command is associated with a test identifier, and wherein the test service is associated with the test identifier.

9. The computer-implemented method of claim 7 further comprising:

collecting additional monitoring information, wherein the additional monitoring information is collected based at least in part on processing at least one additional production storage command;

analyzing the additional monitoring information to identify that the storage traffic overload on the storage processing service continues to exist; and

continuing to process, at the storage processing service, the subsequent production storage commands.

10. The computer-implemented method of claim 9, wherein analyzing the additional monitoring information to identify that the storage traffic overload on the storage processing service continues to exist comprises determining that a completion criterion for assessing whether the storage traffic overload exists is not satisfied, and wherein the completion criterion comprises a threshold production rate corresponding to a rate of processing production storage commands.

11. The computer-implemented method of claim 7 further comprising:

obtaining additional monitoring information, wherein the additional monitoring information is collected based at least in part on processing at least one additional production storage command;

analyzing the additional monitoring information to identify that the storage traffic overload on the storage processing service does not continue to exist; and

resuming processing of test storage commands.

12. The computer-implemented method of claim 6, wherein routing the test storage command to the test service comprises:

receiving an API-level request comprising a test identifier; and

determining that the test identifier indicates that the test storage command is to be processed on the test service and on a second production service associated with the storage processing service.

13. The computer-implemented method of claim 12, wherein routing the test storage command to the test service further comprises:

transmitting routing information indicating that the test storage command is to be processed on the test service and on the second production service; and

transmitting additional routing information indicating that any subsequent production storage commands associated with the second production service are to be processed on the second production service substantially concurrently with the test storage command.

14. The computer-implemented method of claim 6, wherein the storage processing service comprises one or more storage computing devices configured to obtain at least one of the first storage command or the second storage command from a virtual computing device instance, the virtual computing device instance configured to transmit storage commands via a communications network, the virtual computing device instance executing on behalf of a client computing device, and the virtual computing device instance hosted on a host computing device.

15. A non-transitory computer-readable storage medium including computer-executable instructions comprising:

computer-executable instructions that, when executed by a computing device associated with one or more client computing devices, cause the computing device to at least;

obtain, at a production service, a first storage command comprising a first tag indicating that the first storage command is a production storage command, and a second storage command comprising a second tag indicating that the second storage command is a test storage command;

process the production storage command at least partly by characterizing the production storage command using the first tag;

process the test storage command at least partly by characterizing the test storage command using the second tag;

obtain monitoring information regarding processing the production storage command and the test storage command; and

modify the processing of a subsequent test storage command based at least in part on the monitoring information.

16. The non-transitory computer-readable storage medium of claim 15, wherein the production service corresponds to at least one of a storage placement service, a volume metadata service, or a replication service.

17. The non-transitory computer-readable storage medium of claim 15 further comprising computer-executable instructions that, when executed by the computing device associated with the one or more client computing devices, cause the computing device to at least:

receive a routing request for the test storage command from a storage processing service;

determine that the test storage command is associated with a test identifier of an application-level (“

API-level”

) request; and

determine that the test storage command is to be routed to a test service associated with the production service, wherein the test service is configured, in part, by the API-level request.

18. The non-transitory computer-readable storage medium of claim 17, wherein the routing request includes routing information indicating that a test storage volume of the storage processing service is to process the test storage command, wherein the test storage command is associated with the test identifier.

19. The non-transitory computer-readable storage medium of claim 15 further comprising computer-executable instructions that, when executed by the computing device associated with the one or more client computing devices, cause the computing device to at least:

receive a request to route an additional test storage command from the production service; and

transmit additional routing information indicating that the additional test storage command is to be shed from a processing queue of the production service, wherein the test service is associated with the production service.