System and method for performing object-modifying commands in an unstructured storage service

• US 9,760,576 B1
• Filed: 03/17/2014
• Issued: 09/12/2017
• Est. Priority Date: 08/23/2011
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Lee, Kerry Quintin, Denny-Brown, II, Derek Ernest, Marupadi, Varun
• Primary Examiner(s): Waldron, Scott A
• Application Number:
  US14/216,216
• Time in Patent Office: 1,275 Days
• CPC Class Codes: G06F 16/17 Details of further file system functions
• CPC Class Codes: G06F 16/2456 Join operations
• CPC Class Codes: G06F 16/30 of unstructured textual data document management systems G06F16/93
• CPC Class Codes: G06F 16/33 Querying
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G06F 16/4387 by the use of playlists
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G06F 16/835 Query processing
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/93 Document management systems

Abstract

Embodiments of system and method for performing object-modifying commands in an unstructured storage service are described. Embodiments may include utilizing one or more nodes to implement an unstructured storage service including a service interface configured to service data storage requests and data retrieval requests from remote clients. The unstructured storage service is configured to isolate access to each of multiple storage domains of the service to different clients. Embodiments may also include receiving from a given remote client, a request to store a data object to a given storage domain of the service. The request further specifying a command to modify source data for the object prior to storing the object. Embodiments may also include using the unstructured storage service to perform the command without exporting the source data from the unstructured storage service to the given remote client prior to completion of the command.

Claims

1. A computer-implemented method, comprising:

running a data storage service on one or more computer nodes, wherein the data storage service implements an application programming interface for servicing data storage requests and data retrieval requests received over a network from a plurality of remote client devices;

receiving, by the data storage service and from a given remote client device, a request, according to the application programming interface of the storage service, to store a data object to a storage domain of the data storage service or to retrieve a data object from a storage domain of the data storage service, the request further specifying, according to the application programming interface of the data storage service, a computation to modify data of the object prior to storing or subsequent to retrieving the data object in accordance with the request; and

performing, by the data storage service, the computation without exporting the data object from the data storage service to the given remote client device prior to completion of the computation.

2. The method of claim 1, wherein said performing the computation comprises:

determining whether a measure of computational intensity of the computation exceeds a threshold value of computational intensity for the data storage service;

if the measure of computational intensity does not exceed the threshold value, performing the computation within the data storage service without exporting the data object from the data storage service to the given remote client device prior to completion of the computation; and

if the measure of computational intensity exceeds the threshold value, using one or more external components to perform the computation on the data object without exporting the data object from the data storage service to the given remote client device prior to completion of the computation, wherein the one or more external components are external to the data storage service.

3. The method of claim 2, wherein said using one or more external components to perform the computation comprises:

determining whether the measure of computational intensity of the computation exceeds a threshold value of computational intensity for a first external component one of the one or more external components;

if the measure of computational intensity does not exceed the threshold value for first external component, performing the computation using the first external component without exporting the data object from the data storage service to the given remote client device prior to completion of the computation; and

if the measure of computational intensity exceeds the threshold value for the first external component, using one or more additional ones of the external components to perform the computation on the data object without exporting the data object from the data storage service to the given remote client device prior to completion of the computation.

4. The method of claim 2, wherein the one or more external components comprise one or more of:

a virtual instance of a computer system executing on a node outside of the data storage service, or a node cluster configured to perform the computation.

5. The method of claim 2, wherein the one or more external components are configured to access one or more data objects of a storage domain of the data storage service, the method further comprising:

receiving, by the one or more external components, an authentication token granting access to the one or more data objects; and

authenticating, by the one or more external components, with the data storage service using the authentication token prior to accessing the one or more data objects.

6. A system, comprising:

one or more computer nodes comprising at least one hardware computing device and configured to implement a data storage service including a service interface configured to service data storage requests and data retrieval requests from remote client devices;

wherein the data storage service is configured to;

receive, via an application programming interface of the data storage service interface, from a given remote client device, a request to store a data object to a storage domain of the data storage service or to retrieve a data object from a storage domain of the data storage service, the request further specifying a computation to retrieving the object in accordance with the request, wherein the application programming interface is configured to receive requests over a network from a plurality of remote client devices; and

wherein the data storage service is configured to perform the computation without exporting the source data from the data storage service to the given remote client device prior to completion of the computation.

7. The system of claim 6, wherein the data storage service is further configured to:

receive, via the application programming interface of the data storage service, from the given remote client device, a state request for the computation; and

in response to the state request, send a state response to the given remote client device, wherein the state response comprises information indicating a measure of progress for the computation.

8. The system of claim 7, wherein the state response comprises a measure of cost associated with the computation within the data storage service.

9. The system of claim 6, wherein the data storage service is further configured to:

receive, via the application programming interface of the data storage service, from the given remote client device, a request to cancel the computation, wherein said performing the computation is ongoing when the request to cancel is received; and

in response to receiving the request to cancel, stop said performing the computation.

10. The system of claim 9, wherein the storage service is further configured to:

determine a measure of cost associated with the canceled computation based on one or more of;

a quantity of data accessing during said performing the computation or a measure of computational intensity of the computation.

11. The system of claim 9, wherein the request to cancel comprises an operation identifier identifying the computation to cancel.

12. The system of claim 6, wherein the source data includes one or more of:

data from an existing object from within the data storage service, or data provided by the remote client device at the time of the request.

13. The system of claim 6, wherein different storage requests are sent by remote client devices of different customer entities, wherein the data storage service is configured to assign a respective operation identifier to each request.

14. A non-transitory, computer-readable storage medium storing program instructions that when executed on a computing device cause the computing device to implement a data storage service configured to:

provide a service interface for servicing data storage requests and data retrieval requests from remote client devices;

receive, via an application programming interface of the service interface, from a given remote client device, a request to store a data object to a storage domain of the service or retrieve a data object from a storage domain of the data storage service, the request further specifying a computation to modify source data of the object prior to storing or subsequent to retrieving the object in accordance with the request, wherein the application programming interface is configured for servicing requests received over a network from a plurality of remote client devices; and

perform the computation without exporting the data from the data storage service to the given remote client device prior to completion of the computation.

15. The non-transitory, computer-readable storage medium of claim 14, wherein the service interface comprises a web service interface accessible to the remote client devices over the Internet, wherein the web service interface is configured to implement the application programming interface, and wherein the service interface is further configured to implement another service interface accessible to local client devices over an intranet connected to the data storage service.

16. The non-transitory, computer-readable storage medium of claim 14, wherein the data storage service is configured to internally replicate the stored data object, and wherein said internally replicating indicates completion of the computation to the given remote client device.

17. The non-transitory, computer-readable storage medium of claim 14, wherein the request to store an object to a storage domain of the data storage service includes an HTTP PUT computation, wherein the request to retrieve an object from a storage domain of the data storage service includes an HTTP GET computation.

18. The non-transitory, computer-readable storage medium of claim 14, wherein the computation to modify the data includes a computation to perform one or more of:

data compression, encryption, sampling, or random number generation.

19. The non-transitory, computer-readable storage medium of claim 14, wherein the source data includes one or more of:

data from an existing object from within the data storage service, or data provided by the given remote client device at the time of the request.

20. The non-transitory, computer-readable storage medium of claim 14, wherein the service interface is a web service interface accessible to the remote client devices over the Internet, and wherein the web service interface is configured to implement one or more of:

a parameter-based web services architecture, a document-based web services architecture, a message-based web services architecture, a document-encapsulating access protocol, an interface publishing protocol, or an interface discovery protocol.