Query rewind mechanism for processing a continuous stream of data

• US 8,620,945 B2
• Filed: 09/23/2010
• Issued: 12/31/2013
• Est. Priority Date: 09/23/2010
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Chen, Qiming, Hsu, Meichun
• Primary Examiner(s): Corrielus, Jean M
• Assistant Examiner(s): ASPINWALL, EVAN S
• Application Number:
  US12/888,427
• Publication Number:
  US 20120078939A1
• Time in Patent Office: 1,195 Days
• Field of Search: 707/615, 707/648, 707/670, 707/682, 707/684, 707/685, 707/702, 707/703, 707/713, 707/769, 707/826
• US Class Current: 707/769
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

A query engine receives a continuous stream of data and applies a Structured Query Language (SQL) query to the data. The data is processed on a chunk-by-chunk basis with a query rewind mechanism that sustains the SQL query as a single long-standing query that allows a state of the SQL query to be maintained.

Claims

1. A method executed by a computing system, comprising:

receiving, at a query engine in a database system, a continuous stream of data in data chunks;

applying a Structured Query Language (SQL) query with User Defined Functions (UDFs) to the data chunks of the continuous stream of data; and

processing, by the database system, the data chunks on a chunk-by-chunk basis with a query rewind mechanism to sustain the SQL query as a single long-standing query that allows the UDFs and a state of the SQL query to be maintained.

2. The method of claim 1 further comprising:

receiving, at the query engine, an end-of-data message that instructs the query engine to terminate execution of the SQL query; and

executing the query rewind mechanism in response to receiving the end-of-data message.

3. The method of claim 1 further comprising:

activating the query rewind mechanism; and

reactivating a stream source function for processing newly incoming data in the continuous stream of data in response to activating the query rewind mechanism.

4. The method of claim 1 further comprising, joining two continuous streams of data where the query rewind mechanism serves as a synchronization point that processes the two continuous streams of data.

5. The method of claim 1, wherein static data in the UDFs are loaded only once rather than repeatedly fetching the static data in a cycle-by-cycle manner.

6. A tangible, non-transitory computer readable storage medium storing instructions that, when executed by a database system, cause the database system to:

receiving, at a query engine, a continuous stream of data;

processing, at the query engine, a Structured Query Language (SQL) query on the continuous stream of data on a chunk-by-chunk basis; and

activating a query rewind mechanism to sustain the SQL query as a single long-standing query.

7. The tangible, non-transitory computer readable storage medium of claim 6, wherein the SQL query is the single long-standing query as opposed to multiple queries that are repeatedly launched on windows of data.

8. The tangible, non-transitory computer readable storage medium of claim 6 further comprising, capturing the continuous stream of data with stream source functions, wherein the stream source functions are kind of User Defined Function that is extended with support from the query engine.

9. The tangible, non-transitory computer readable storage medium of claim 6, wherein the query engine continuously analyzes the continuous stream of data, as opposed to storing the data on a disk, retrieving the data from the disk, and subsequently analyzing the data.

10. The tangible, non-transitory computer readable storage medium of claim 6 wherein the query rewind mechanism iteratively rewinds the SQL query for processing the continuous stream of data in time windows.

11. A computing system, comprising:

a database;

a query engine in communication with the database, the query engine receives a continuous stream of data and a query; and

a query rewind mechanism that processes, on a chunk-by-chunk basis, the continuous stream of data as a single long-standing query.

12. The computing system of claim 11, wherein the query is applied to the continuous stream of data on a chunk-by-chunk basis as opposed to being applied to an entire input relation of the continuous stream of data.

13. The computing system of claim 11, wherein the query is sustained across multiple running cycles and data from the query is stored in User Defined Functions (UDFs) over the multiple running cycles such that states of the UDFs are maintained continuously regardless of cycle boundaries.

14. The computing system of claim 11, wherein an end-of-data message terminates execution of the query and cuts the continuous stream of data into chunks that are based on a window of time.

15. The computing system of claim 11, wherein the query rewind mechanism rewinds the query for processing data falling in a subsequent window of time.