Command aggregation for data storage operations

• US 10,810,157 B1
• Filed: 09/28/2016
• Issued: 10/20/2020
• Est. Priority Date: 09/28/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Paterra, Frank Charles, Neilsen, Eric, Basarir, Firat, Portnoy, Andrew
• Primary Examiner(s): Meng, Jau Shya
• Application Number:
  US15/279,225
• Time in Patent Office: 1,483 Days
• Field of Search: None
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/14 Details of searching files based on file metadata
• CPC Class Codes: G06F 16/168 Details of user interfaces specifically adapted to file systems, e.g. browsing and visualisation, 2d or 3d GUIs query results presentation G06F16/156
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: G06F 9/547 Remote procedure calls [RPC]; Web services

Abstract

A data storage system implements aggregation, bifurcation, and/or reduction techniques to improve the efficiency of processing data storage requests. Data storage requests and/or their associated payloads may be aggregated based on one or more parameters. Data to be stored and the associated commands may be separated so as to optimize a system'"'"'s throughput and latency for each. Furthermore, extraneous commands and requests may be reduced or eliminated based on heuristics associated with the requests and the data.

Claims

1. A computer-implemented method, comprising:

receiving, at a server that includes one or more processors, from a client entity, a programmatic request to store data on data storage of the server, the data storage including a file system, the programmatic request including the data and a plurality of programmatic subrequests aggregated by the client entity and transmitted by the client entity to the server as an aggregated request, the plurality of programmatic subrequests selected at least based on the programmatic request triggering at least one additional request related to the programmatic request to store the data on the data storage of the server;

determining, from the file system, one or more file system commands resulting from a storage command associated with the programmatic request, fulfillment of the one or more file system commands being a precondition of executing the storage command;

executing the one or more file system commands based at least in part on metadata associated with the plurality of programmatic subrequests; and

executing the storage command to store the data on the data storage.

2. The computer-implemented method of claim 1, wherein the plurality of programmatic subrequests are generated from the programmatic request based at least in part on a type of the file system.

3. The computer-implemented method of claim 1, wherein the execution of the one or more file system commands results in an exception provided to the client entity as a result of the plurality of programmatic subrequests including insufficient or incorrect metadata.

4. The computer-implemented method of claim 1, wherein the programmatic request and the plurality of programmatic subrequests are API calls.

5. A system, comprising:

one or more processors; and

memory storing computer-executable instructions that, as a result of being executed by the one or more processors, cause the system to;

receive a programmatic request to perform a file system command, the programmatic request including a plurality of programmatic subrequests based on at least one additional command to be triggered when the file system command is executed, the plurality of programmatic subrequests combined with the programmatic request by a client entity and transmitted by the client entity as an aggregated request to the system; and

process the programmatic request by at least;

executing the file system command;

determining one or more additional file system commands based at least in part on executing the file system command; and

executing the one or more additional file system commands in connection with the file system command at least in part using metadata associated with one or more of the plurality of programmatic subrequests.

6. The system of claim 5, wherein the programmatic request is an API call.

7. The system of claim 5, wherein the metadata is derived from a policy associated with the programmatic request.

8. The system of claim 5, wherein the file system command is a data write command.

9. The system of claim 5, wherein the system further ignores a portion of the metadata associated with the plurality of programmatic subrequests.

10. The system of claim 5, wherein the system further generates an additional programmatic request related to the file system command based at least in part on the file system command and at least a portion of the metadata.

11. The system of claim 5, wherein the programmatic request includes data associated with the file system command.

12. The system of claim 5, wherein the system further provides an outcome of the file system command in response to the programmatic request.

13. A non-transitory computer-readable storage medium comprising executable instructions that, when executed by one or more processors of a computer system, cause the computer system to at least:

associate a programmatic request to perform a file system operation with a file system command, the programmatic request including metadata, associated with one or more additional file system commands to occur as a result of executing the file system command, the metadata aggregated with the programmatic request by a client entity and delivered by the client entity to the computer system as one payload; and

execute the file system operation associated with the programmatic request by at least;

executing the file system command; and

executing the one or more additional file system commands in connection with the file system command using at least a subset of the metadata.

14. The non-transitory computer-readable storage medium of claim 13, wherein the metadata includes one or more programmatic subrequests related to the programmatic request.

15. The non-transitory computer-readable storage medium of claim 13, wherein the file system command is associated with a new file system location.

16. The non-transitory computer-readable storage medium of claim 15, wherein at least some of the one or more additional file system commands includes a command to create the new file system location in connection with executing the file system command.

17. The non-transitory computer-readable storage medium of claim 13, wherein at least a subset of the metadata includes information to generate the one or more additional file system commands.

18. The non-transitory computer-readable storage medium of claim 13, wherein an outcome of the programmatic request is provided to a requestor via an entity through which the programmatic request is received.

19. The non-transitory computer-readable storage medium of claim 13, wherein an exception resulting from processing the programmatic request is provided to an entity through which the programmatic request is received.

20. The non-transitory computer-readable storage medium of claim 13, wherein the programmatic request is a web service call.