Continuous Querying of a Data Stream

• US 20130191413A1
• Filed: 10/14/2010
• Published: 07/25/2013
• Est. Priority Date: 10/14/2010
• Status: Active Grant

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Inventors: Chen, Qiming, Hsu, Meichun
• Granted Patent: US 9,195,708 B2
• Time in Patent Office: Days
• US Class Current: 707/769
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

In continuous querying of a data stream, a query including query cycles can be initialized (310) on a query engine to analyze the data stream for desired information. The data stream can be processed (320) as segments, where a size of the segments is based on a user-defined parameter. The query cycles can be synchronized (330) with the segments of the data stream. A first segment can be analyzed (340) by performing the query on the first segment to obtain a first result. A query state of the query can be persisted (350) and the query operation can be rewound to begin a new query cycle. A second segment can be analyzed (360) in the new query cycle by performing the query on the second segment based on the first result.

Claims

1. An article of manufacture including a computer usable medium having computer readable program code embodied therein for continuous querying of a data stream, comprising computer readable program code capable of performing the operations of:

initializing (310) a query including query cycles on a query engine to query the data stream for desired information;

processing (320) the data stream as segments, wherein a size of the segments is based on a defined parameter;

synchronizing (330) the query cycles with the segments of the data stream;

analyzing (340) a first segment in a first computer memory by performing the query on the first segment using a processor to obtain a first result;

persisting (350) a query state of the query and rewinding a query operation to begin a new query cycle; and

analyzing (360) a second segment in a second computer memory in the new query cycle by performing the query on the second segment using the processor and based on the first result.

2. An article of manufacture as in claim 1, wherein the computer readable program code is further capable of providing substantially real-time notification of receipt of a segment.

3. An article of manufacture as in claim 1, wherein the computer readable program code is further capable of receiving a second segment and wherein analyzing the first segment is performed as the second segment is received.

4. An article of manufacture as in claim 1, wherein analyzing the second segment based on the first result comprises performing history-sensitive continuous analytics.

5. An article of manufacture as in claim 1, wherein the computer readable program code is further capable of sustaining a buffer across query cycles to provide a sliding query window.

6. An article of manufacture as in claim 1, wherein the computer readable program code is further capable of computing structured query language (SQL) aggregate functions for each segment without persisting context of the aggregate functions from one segment to the next.

7. An article of manufacture as in claim 1, wherein the first computer memory and the second computer memory respectively comprise a first table and a second table, the method further comprising:

truncating the first table after analyzing the second segment; and

analyzing a third segment in the first table.

8. An article of manufacture as in claim 1, wherein the first computer memory and the second computer memory respectively comprise a first table and a second table, the first table comprising a write table and the second table comprising a read table, and wherein the computer readable program code is further capable of:

swapping the names of the first table and the second table using data dictionary manipulation, said swapping comprising;

renaming the write table to be the read table; and

renaming the read table to be the write table.

9. An article of manufacture as in claim 8, wherein swapping comprises swapping the names without copying or moving contents of the first table and second table.

10. A system (100) for continuous querying of a data stream, comprising:

a query engine (170) for analyzing a continuous data stream for desired information;

an analysis server (120) for delivering the continuous data stream to the query engine in data segments;

a query (180) executing on the query engine in cycles synchronized with the data segments of the continuous data stream;

a first buffer (140) for receiving a first data segment;

a second buffer (150) for receiving a second data segment; and

a buffer pool management module (130) for managing alternately reading and writing from the first buffer and the second buffer.

11. A system as in claim 10, wherein the first buffer comprises a write buffer and the second buffer comprises a read buffer, and the buffer pool management module swaps the names of the first buffer and the second buffer using data dictionary manipulation by renaming the first buffer to be a read buffer and the second buffer to be a write buffer.

12. A system as in claim 11, wherein the buffer pool management module swaps the names of the first buffer and the second buffer without copying or moving contents of the first buffer and second buffer.

13. A system as in claim 10, wherein the query comprises a self-referencing, history-sensitive query, and wherein the query engine continuously and alternately queries the first buffer and the second buffer such that the first buffer is analyzed while the second buffer receives the second data segment and the second buffer is analyzed while the first buffer receives a third data segment.

14. A system as in claim 10, further comprising a notification module (160) for providing substantially real time notification of receipt of a segment to the query engine to begin executing the query on the segment.

15. A method (400) for continuous querying of a data stream, comprising:

receiving (410) a data stream from a data source;

executing (420) a cycle-based continuous query on the segments of the data stream;

delivering (430) analytics results from execution of the cycle-based continuous query to two destination tables, alternately cycle-by-cycle;

simultaneously producing and consuming (440) the analytics results by producing the results at one of the two destination tables while consuming the analytics results at another one of the two destination tables; and

persisting (450) a query state of the cycle-based continuous query across segments and rewinding a query operation to begin a new query cycle with the same cycle-based continuous query.