PROCESSING PRE-EXISTING DATA SETS AT AN ON DEMAND CODE EXECUTION ENVIRONMENT

• US 20170286143A1
• Filed: 03/30/2016
• Published: 10/05/2017
• Est. Priority Date: 03/30/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Wagner, Timothy Allen, Brooker, Marc John, Nair, Ajay
• Granted Patent: US 10,891,145 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 12/0875 with dedicated cache, e.g. instruction or stack
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 2212/1016 Performance improvement
• CPC Class Codes: G06F 2212/283 Plural cache memories
• CPC Class Codes: G06F 2212/452 Instruction code
• CPC Class Codes: G06F 2212/453 Microcode or microprogram
• CPC Class Codes: G06F 9/45516 Runtime code conversion or optimisation
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

Systems and methods are described for transforming a data set within a data source into a series of task calls to an on-demand code execution environment or other distributed code execution environment. Such environments utilize pre-initialized virtual machine instances to enable execution of user-specified code in a rapid manner, without delays typically caused by initialization of the virtual machine instances, and are often used to process data in near-real time, as it is created. However, limitations in computing resources may inhibit a user from utilizing an on-demand code execution environment to simultaneously process a large, existing data set. The present application provides a task generation system that can iteratively retrieve data items from an existing data set and generate corresponding task calls to the on-demand computing environment, while ensuring that at least one task call for each data item within the existing data set is made.

Claims

1. A system for processing a plurality of data items within a data source via an on-demand code execution environment, the system comprising:

a non-transitory data store configured to implement;

an in-process data cache indicating data items, from the plurality of data items, that has been identified by the system but not yet processed at the on-demand code execution environment; and

a results data cache indicating data items, from the plurality of data items, that have been processed at the on-demand code execution environment;

one or more processors configured to implement a user interface subsystem that obtains, from a user computing device, information identifying the data source and a task, on the on-demand code execution environment, to utilize in processing the plurality of data items;

one or more processors configured to implement a data retrieval subsystem that;

retrieves a first set of data items, from the plurality of data items, from the data source; and

for data items of the set of data items;

generates an identifier for the data item;

determines, from the identifier, that the data item is not identified within the in-process data cache or the results data cache; and

enqueues the data item in the in-process data cache;

one or more processors configured to implement a call generation subsystem that;

identifies one or more data items from the in-process data cache;

submits a call to the on-demand code execution environment to execute the task to process the one or more data items;

determines that the task successfully processed the one or more data items; and

places the one or more data items in the results data cache;

wherein the user interface subsystem further transmits a notification to the user computing device when the plurality of data items have been processed at the on-demand code execution environment.

2. The system of claim 1, wherein the one or more processors implementing the user interface system, the one or more processors implementing the data retrieval subsystem, and the one or more processors implementing the call generation subsystem are the same one or more processors.

3. The system of claim 1, wherein the data source corresponds to a plurality of distinct data sources.

4. The system of claim 1, wherein the task corresponds to a plurality of tasks, and wherein the information obtained from the user computing device further comprises information for determining an individual task, from the plurality of tasks, to call for processing a given data item from the plurality of data items.

5. The system of claim 1, wherein the data retrieval system is configured to generate the identifier for the data item by at least one of identifying a key of the data item within the data source, concatenating one or more attributes of the data item, or processing at least a portion of the data item according to a hash function.

6. The system of claim 1, wherein the one or more processors implementing the call generation subsystem place the one or more data items in the results data cache by listing a pointer to the one or more data items within the results data cache.

7. A computer-implemented method comprising:

maintaining a data cache indicating data items, from a plurality of data items within a data source, that have been identified for processing at an on-demand code execution environment;

iteratively retrieving data items, from the plurality of data items, from the data source;

enqueing within the data cache those retrieved data items, from the set of data items, that are not currently identified within the data cache;

iteratively submitting calls to the on-demand code execution environment to process data items from the data cache by execution of a task;

recording results of the calls submitted with respect to the data items from the data cache;

determining that no additional data items, from the plurality of data items, are awaiting processing by execution of the task; and

transmitting, to a user computing device associate with the plurality of data items, a notification indicating that processing of the plurality of data items has been completed.

8. The computer-implemented method of claim 7, wherein enqueing within the data cache those retrieved data items, from the set of data items, that are not currently identified within the data cache comprises associating identifiers of the enqueued data items with the data cache.

9. The computer-implemented method of claim 7 further comprising generating identifiers for the retrieved data items.

10. The computer-implemented method of claim 9 further comprising identifying those retrieved data items, from the set of data items, that are not currently identified within the data cache at least partly by comparing identifiers of the retrieved data items to identifiers currently associated with the data cache.

11. The computer-implemented method of claim 7 further comprising obtaining information identifying the data source and the task from a user computing device.

12. The computer-implemented method of claim 11, wherein the information obtained from the user computing device further includes a maximum rate for at least one of retrieval of data items from the data source or submission of calls to the on-demand code execution environment.

13. The computer-implemented method of claim 7, wherein iteratively submitting calls to the on-demand code execution environment to process data items from the data cache by execution of a task and recording results of the calls submitted with respect to the data items from the data cache occurs at least partly contemporaneously to iteratively retrieving data items, from the plurality of data items, from the data source and enqueing within the data cache those retrieved data items, from the set of data items, that are not currently identified within the data cache.

14. The computer-implemented method of claim 7 further comprising determining a call submission rate for the on-demand code execution environment based at least in part on errors received from the on-demand code execution environment.

15. The computer-implemented method of claim 14 further comprising modifying a rate at which data items are iteratively retrieved from the data source based on the call submission rate.

16. The computer-implemented method of claim 14 further comprising:

determining that a retrieved data item represents a modification of a prior data item that is currently identified within the data cache;

enqueuing the retrieved data item within the data cache;

submitting a call to the on-demand code execution environment to process the retrieved data item; and

overwriting a result of a call submitted with respect to the prior data item with a result of the retrieved data item.

17. The computer-implemented method of claim 16, wherein the call to the call to the on-demand code execution environment to process the retrieved data item is a call to execute a second task associated with modified data items.

18. Non-transitory computer readable media including computer-executable instructions, wherein the computer-executable instructions, when executed by a computing system, cause the computing system to:

maintain a data cache indicating data items, from a plurality of data items within a data source, that have been identified for processing at an on-demand code execution environment;

retrieve data items, from the plurality of data items, from the data source;

enqueue within the data cache those retrieved data items, from the set of data items, that are not currently identified within the data cache;

submit calls to the on-demand code execution environment to process data items from the data cache by execution of a task;

record results of the calls submitted with respect to the data items from the data cache;

determine that no additional data items, from the plurality of data items, are awaiting processing by execution of the task; and

transmit, to a user computing device associate with the plurality of data items, a notification indicating that processing of the plurality of data items has been completed.

19. The non-transitory computer-readable media of claim 18, wherein the computer-executable instructions further cause the computing system to:

identify one or more additional data items from the data cache;

mark the one or more additional data items within the data cache with an in-flight indicator;

submit a call to the on-demand code execution environment to process the one or more additional data items by execution of a task;

determine that the task did not successfully process the one or more additional data items; and

remove the in-flight indicator from the one or more additional data items.

20. The non-transitory computer-readable media of claim 18, wherein the computer-executable instructions further cause the computing system to generate identifiers for the retrieved data items.

21. The non-transitory computer-readable media of claim 18, wherein the computer-executable instructions further cause the computing system to identifying those retrieved data items, from the set of data items, that are not currently identified within the data cache at least partly by comparing identifiers of the retrieved data items to identifiers currently associated with the data cache.

22. The non-transitory computer-readable media of claim 18, wherein the computer-executable instructions cause the computing system to retrieve data items from the data source iteratively.

23. The non-transitory computer-readable media of claim 18, wherein the computer-executable instructions cause the computing system to enqueue data items within the data cache by recording an identifier of the data item within the data cache.