INTEGRATING EVENT PROCESSING WITH MAP-REDUCE

• US 20140156683A1
• Filed: 11/13/2013
• Published: 06/05/2014
• Est. Priority Date: 12/05/2012
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: de Castro Alves, Alexandre
• Granted Patent: US 10,956,422 B2
• Time in Patent Office: Days
• US Class Current: 707/755
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

Large quantities of data can be processed and/or queried relatively quickly using a combination of continuous event processing and a Map-Reduce algorithmic tool. The continuous event processor can continuously produce real-time results by merging (a) CQL query results from events received since a currently executing Map-Reduce job was started with (b) a most recent query result produced by a most recently completed Map-Reduce job. When the currently executing Map-Reduce job completes, its query result can be stored and made accessible to the continuous event processor, and a new Map-Reduce job can be started relative to event data that has grown in size since the execution of the last Map-Reduce job. The Map-Reduce algorithmic tool provides a convenient mechanism for analyzing and processing large quantities of data.

Claims

1. A computer-implemented method comprising:

starting a first batch operation that executes a first query relative to event data that has been stored in a persistent data store as of a first time;

while the first batch operation is executing, continuously executing a second query relative to event data that is continuously being received from a data stream;

while the first batch operation is executing, continuously merging results of the second query with a result of a second batch operation that has completed;

continuously outputting results of the merging; and

in response to a completion of the first batch operation, replacing the result of the second batch operation with a result of the first batch operation.

2. The computer-implemented method of claim 1, further comprising:

while the first batch operation is executing, continuously storing, in the persistent data store, the event data that is continuously being received from the data stream;

in response to the completion of the first batch operation, starting a third batch operation that executes the first query relative to event data that has been stored in the persistent data store as of a second time that is later than the first time;

while the third batch operation is executing, continuously executing the second query relative to the event data that is continuously being received from a data stream;

while the third batch operation is executing, continuously merging results of the second query with a result of the first batch operation that has completed;

continuously outputting results of the merging of the second query with the result of the first batch operation that has completed; and

in response to a completion of the third batch operation, replacing the result of the first batch operation with a result of the third batch operation.

3. The computer-implemented method of claim 1, wherein the first batch operation is a Map-Reduce operation, and wherein the second query performs same operations as the first query.

4. The computer-implemented method of claim 1, further comprising:

receiving a continuous query language (CQL) query as the second query; and

automatically generating, based on the CQL query, a Map-Reduce program that performs operations of the CQL query.

5. The computer-implemented method of claim 1, further comprising:

receiving a continuous query language (CQL) query as the second query;

automatically parsing the CQL query;

based on the parsing, automatically separating the CQL query into a first set of operators;

automatically generating, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators; and

automatically generating, based on the Pig Latin operators, a Map-Reduce program that performs the first batch operation.

6. The computer-implemented method of claim 1, further comprising:

receiving a continuous query language (CQL) query as the second query;

automatically parsing the CQL query;

based on the parsing, automatically separating the CQL query into a first set of operators;

automatically generating, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators; and

automatically generating a directed graph of nodes including operators from the first set, operators from the second set, edges from nodes representing operators from the first set to nodes representing operators from the second set, and edges from nodes representing operators form the second set to operators from the first set.

7. The computer-implemented method of claim 1, further comprising:

receiving a continuous query language (CQL) query as the second query;

automatically parsing the CQL query;

based on the parsing, automatically separating the CQL query into a first set of operators;

automatically generating, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators;

automatically generating a directed graph of nodes including operators from the first set and operators from the second set; and

automatically merging two or more nodes in the directed graph based on similarities between operations performed by the two or more nodes.

8. A computer-readable memory storing particular instructions to cause one or more processors to perform operators, the particular instructions comprising:

instructions to cause the one or more processors to start a first batch operation that executes a first query relative to event data that has been stored in a persistent data store as of a first time;

instructions to cause the one or more processors to continuously execute, while the first batch operation is executing, a second query relative to event data that is continuously being received from a data stream;

instructions to cause the one or more processors to continuously merge, while the first batch operation is executing, results of the second query with a result of a second batch operation that has completed;

instructions to cause the one or more processors to continuously output results of the merging; and

instructions to cause the one or more processors to replace, in response to a completion of the first batch operation, the result of the second batch operation with a result of the first batch operation.

9. The computer-readable memory of claim 8, wherein the particular instructions further comprise:

instructions to cause the one or more processors to continuously store, while the first batch operation is executing, and in the persistent data store, the event data that is continuously being received from the data stream;

instructions to cause the one or more processors to start, in response to the completion of the first batch operation, a third batch operation that executes the first query relative to event data that has been stored in the persistent data store as of a second time that is later than the first time;

instructions to cause the one or more processors to continuously execute, while the third batch operation is executing, the second query relative to the event data that is continuously being received from a data stream;

instructions to cause the one or more processors to continuously merge, while the third batch operation is executing, results of the second query with a result of the first batch operation that has completed;

instructions to cause the one or more processors to continuously output results of the merging of the second query with the result of the first batch operation that has completed; and

instructions to cause the one or more processors to replace, in response to a completion of the third batch operation, the result of the first batch operation with a result of the third batch operation.

10. The computer-readable memory of claim 8, wherein the first batch operation is a Map-Reduce operation, and wherein the second query performs same operations as the first query.

11. The computer-readable memory of claim 8, wherein the particular instructions further comprise:

instructions to cause the one or more processors to receive a continuous query language (CQL) query as the second query; and

instructions to cause the one or more processors to automatically generate, based on the CQL query, a Map-Reduce program that performs operations of the CQL query.

12. The computer-readable memory of claim 8, wherein the particular instructions further comprise:

instructions to cause the one or more processors to receive a continuous query language (CQL) query as the second query;

instructions to cause the one or more processors to automatically parse the CQL query;

instructions to cause the one or more processors to automatically separate, based on the parsing, the CQL query into a first set of operators;

instructions to cause the one or more processors to automatically generate, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators; and

instructions to cause the one or more processors to automatically generate, based on the Pig Latin operators, a Map-Reduce program that performs the first batch operation.

13. The computer-readable memory of claim 8, wherein the particular instructions further comprise:

instructions to cause the one or more processors to receive a continuous query language (CQL) query as the second query;

instructions to cause the one or more processors to automatically parse the CQL query;

instructions to cause the one or more processors to automatically separate, based on the parsing, the CQL query into a first set of operators;

instructions to cause the one or more processors to automatically generate, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators; and

instructions to cause the one or more processors to automatically generate a directed graph of nodes including operators from the first set, operators from the second set, edges from nodes representing operators from the first set to nodes representing operators from the second set, and edges from nodes representing operators form the second set to operators from the first set.

14. The computer-readable memory of claim 8, wherein the particular instructions further comprise:

instructions to cause the one or more processors to receive a continuous query language (CQL) query as the second query;

instructions to cause the one or more processors to automatically parse the CQL query;

instructions to cause the one or more processors to automatically separate, based on the parsing, the CQL query into a first set of operators;

instructions to cause the one or more processors to automatically generate, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators;

instructions to cause the one or more processors to automatically generate a directed graph of nodes including operators from the first set and operators from the second set; and

instructions to cause the one or more processors to automatically merge two or more nodes in the directed graph based on similarities between operations performed by the two or more nodes.

15. A system comprising:

one or more processors; and

a computer-readable memory storing particular instructions to cause the one or more processors to perform operators, the particular instructions comprising;

instructions to cause the one or more processors to start a first batch operation that executes a first query relative to event data that has been stored in a persistent data store as of a first time;

instructions to cause the one or more processors to continuously execute, while the first batch operation is executing, a second query relative to event data that is continuously being received from a data stream;

instructions to cause the one or more processors to continuously merge, while the first batch operation is executing, results of the second query with a result of a second batch operation that has completed;

instructions to cause the one or more processors to continuously output results of the merging; and

instructions to cause the one or more processors to replace, in response to a completion of the first batch operation, the result of the second batch operation with a result of the first batch operation.

16. The system of claim 15, wherein the particular instructions further comprise:

instructions to cause the one or more processors to continuously store, while the first batch operation is executing, and in the persistent data store, the event data that is continuously being received from the data stream;

instructions to cause the one or more processors to start, in response to the completion of the first batch operation, a third batch operation that executes the first query relative to event data that has been stored in the persistent data store as of a second time that is later than the first time;

instructions to cause the one or more processors to continuously execute, while the third batch operation is executing, the second query relative to the event data that is continuously being received from a data stream;

instructions to cause the one or more processors to continuously merge, while the third batch operation is executing, results of the second query with a result of the first batch operation that has completed;

instructions to cause the one or more processors to continuously output results of the merging of the second query with the result of the first batch operation that has completed; and

instructions to cause the one or more processors to replace, in response to a completion of the third batch operation, the result of the first batch operation with a result of the third batch operation.

17. The system of claim 15, wherein the first batch operation is a Map-Reduce operation, and wherein the second query performs same operations as the first query.

18. The system of claim 15, wherein the particular instructions further comprise:

instructions to cause the one or more processors to receive a continuous query language (CQL) query as the second query; and

instructions to cause the one or more processors to automatically generate, based on the CQL query, a Map-Reduce program that performs operations of the CQL query.

19. The system of claim 15, wherein the particular instructions further comprise:

instructions to cause the one or more processors to receive a continuous query language (CQL) query as the second query;

instructions to cause the one or more processors to automatically parse the CQL query;

instructions to cause the one or more processors to automatically separate, based on the parsing, the CQL query into a first set of operators;

instructions to cause the one or more processors to automatically generate, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators; and

instructions to cause the one or more processors to automatically generate, based on the Pig Latin operators, a Map-Reduce program that performs the first batch operation.

20. The system of claim 15, wherein the particular instructions further comprise:

instructions to cause the one or more processors to receive a continuous query language (CQL) query as the second query;

instructions to cause the one or more processors to automatically parse the CQL query;

instructions to cause the one or more processors to automatically separate, based on the parsing, the CQL query into a first set of operators;

instructions to cause the one or more processors to automatically generate, based on specified mappings between CQL operators and Pig Latin operators, a second set of Pig Latin operators that perform operators performed by the first set of CQL operators; and

instructions to cause the one or more processors to automatically generate a directed graph of nodes including operators from the first set, operators from the second set, edges from nodes representing operators from the first set to nodes representing operators from the second set, and edges from nodes representing operators form the second set to operators from the first set.