ONLINE BATCH EXECUTION

• US 20100042998A1
• Filed: 08/14/2008
• Published: 02/18/2010
• Est. Priority Date: 08/14/2008
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: MEKA, Ravikiran, ABDELNUR, Alejandro, BAGRI, Abhijit
• Granted Patent: US 8,181,178 B2
• Time in Patent Office: Days
• US Class Current: 718/101
• CPC Class Codes: G06F 9/4843 by program, e.g. task dispatcher, supervisor, operating system

Abstract

Online batch processing. A job request is received from a user for processing . The job request includes a job configuration and a plurality of operations to process the data. The job configuration is extracted from the job request and stored in a configuration cache. A metadata configuration code is extracted from the job configuration and stored in a code cache. A runtime configuration code is extracted from the job configuration and stored in an instance cache. This allows information to be obtained from the configuration cache, the code cache and the instance cache for processing subsequent job requests with the similar job configuration and the plurality of operations. The data is fetched from at least one of the job request and an external storage device. The plurality of operations is executed on the data to generate a result. The result is provided to the user through at least one of an output stream and the external storage device.

Claims

1. A method for processing data, the method comprising:

receiving a job request from a user, the job request comprises a job configuration and a plurality of operations to process the data;

extracting the job configuration from the job request and storing in a configuration cache;

extracting a metadata configuration code from the job configuration and storing in a code cache;

extracting a runtime configuration code from the job configuration and storing in an instance cache, whereby information can be obtained from the configuration cache, the code cache and the instance cache for processing subsequent job requests with similar job configurations and operations;

fetching the data to be processed from at least one of the job request and an external storage device thereby enabling interoperability between batch execution and online execution;

executing the plurality of operations on the data to generate a result; and

providing the result to the user through at least one of an output stream and the external storage device.

2. The method of claim 1, further comprising:

storing the data in an in-memory datastructure, wherein the in-memory datastructure comprises at least one of a plurality of data and metadata; and

loading the runtime configuration code from the instance cache into the in-memory datastructure.

3. The method of claim 2, wherein executing comprises:

executing the plurality of operations selectively on the data in the in-memory datastructure.

4. The method of claim 2, wherein executing comprises:

executing the plurality of operations on the data in the in-memory datastructure.

5. The method of claim 2, wherein executing comprises:

executing the plurality of operations on metadata in the in-memory datastructure.

6. A method for processing data, the method comprising:

receiving a first job request from a user, the job request comprises a job configuration and a plurality of operations to process the data;

extracting the job configuration from the job request and storing in a configuration cache;

extracting a metadata configuration code from the job configuration and storing in a code cache;

extracting a runtime configuration code from the job configuration and storing in an instance cache;

receiving a second job request from the user, the second job request comprising the same job configuration and plurality of operations as the first job request;

loading the runtime configuration code from the instance cache of the first job request;

fetching the data from at least one of the second job request and an external storage device thereby enabling interoperability between batch execution and online execution;

executing the plurality of operations on the data of the second job request to generate a result; and

providing the result to the user through at least one of an output stream and the external storage device.

7. The method of claim 6 further comprising:

storing the data in an in-memory datastructure, wherein the in-memory datastructure comprises at least one of a plurality of data and metadata.

8. The method of claim 7, wherein executing comprises:

executing the plurality of operations selectively on the data in the in-memory datastructure.

9. The method of claim 7, wherein executing comprises:

executing the plurality of operations on the data in the in-memory datastructure.

10. The method of claim 7, wherein executing comprises:

executing the plurality of operations on metadata in the in-memory datastructure.

11. A computer that runs batch software in an on-line mode, the computer comprising:

a central processor;

a random-access memory including a configuration cache, a code cache and an instance cache;

an input device in communication with the central processor, through which a job request can be communicated to the central processor;

software stored in the random-access memory, operative to cause the central processor to;

extract a metadata configuration and a runtime configuration from the job request and store both configurations in the configuration cache;

extract metadata configuration code from the job request and store said code in the code cache;

extract runtime configuration code from the job request and store said code in the instance cache; and

receive any data in the job request and any data from an external data.

storage device and process the received data to output data according to code stored in the code and instance caches; and

an output device in communication with the central processor, through which the output data can be communicated to a user.

12. The system of claim 11, wherein the output device serializes and sends the output data on an output stream.

13. The system of claim 11, wherein the output device stores the output data into the external storage device, if the data is selected from the external storage device.

14. A machine-readable medium comprising:

a physical medium for storing computer instructions; and

instructions carried by the medium and operable to cause a programmable central processor to perform;

receiving a job request from a user, the job request comprises a job configuration and a plurality of operations to process the data;

extracting the job configuration from the job request and storing in a configuration cache;

extracting a metadata configuration code from the job configuration and storing in a code cache;

extracting a runtime configuration code from the job configuration and storing in an instance cache, whereby information can be obtained from the configuration cache, the code cache and the instance cache for processing subsequent job requests with the similar job configuration and the plurality of operations;

fetching the data from at least one of the job request and an external storage device, thereby enabling interoperability between batch execution and online execution;

executing the plurality of operations on the data to generate a result; and

providing the result to the user through at least one of an output stream and the external storage device.

15. The method of claim 14, further comprising:

storing the data in an in-memory datastructure, wherein the in-memory datastructure comprises at least one of a plurality of data and metadata;

loading the runtime configuration code from the instance cache into the in-memory datastructure.

16. The method of claim 15, wherein executing comprises:

executing the plurality of operations selectively on the data in the in-memory datastructure.

17. The method of claim 15, wherein executing comprises:

executing the plurality of operations on the data in the in-memory datastructure.

18. The method of claim 15, wherein executing comprises:

executing the plurality of operations on metadata in the in-memory datastructure.