Stream Processing by a Query Engine

• US 20120078868A1
• Filed: 09/23/2010
• Published: 03/29/2012
• Est. Priority Date: 09/23/2010
• Status: Active Grant

• Current Assignee: Snowflake, Inc.
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Chen, Qiming, Hsu, Meichun
• Granted Patent: US 8,661,014 B2
• Time in Patent Office: Days
• US Class Current: 707/706
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

A database system executes a method that receives, at a query engine in the database system, a continuous stream of data. The query engine continuously analyzes the continuous stream of data with window functions in a single long-standing query.

Claims

1. ) A method executed by a database system, comprising:

receiving, at a query engine in the database system, a continuous stream of data; and

continuously analyzing, inside the query engine, the continuous stream of data with window functions in a single long-standing query.

2. ) The method of claim 1, wherein the query engine continuously analyses the continuous stream of data as opposed to first storing the continuous stream of data on a disk and then analyzing the continuous stream of data.

3. ) The method of claim 1 further comprising, pushing down processing of the continuous stream of data to a management layer where the query engine executes an application requesting a query of the continuous stream of data using User Defined Functions (UDFs) and performs stream processing of the continuous stream of data.

4. ) The method of claim 1 further comprising, issuing the single long-standing query only once to continuously analyze the continuous stream of data as opposed to repeatedly issuing a query on the continuous stream of data.

5. ) The method of claim 1 further comprising, building the window functions for the query engine to process the continuous stream of data, wherein the window functions keep a history of a data state in a window of time.

6. ) A tangible computer readable storage medium storing instructions that, when executed by a database system, cause the database system to:

receive, at the database system, a continuous stream of data;

issue a single long-standing query to process the continuous stream of data inside a query engine located in the database system; and

process, inside the query engine, the continuous stream of data with the single long-standing query.

7. ) The tangible computer readable storage medium of claim 6, wherein the query engine analyzes the continuous stream of data in real-time before the continuous stream of data is stored in a database.

8. ) The tangible computer readable storage medium of claim 6, wherein the database system further executes the instructions to:

schedule window operations to process the continuous stream of data, wherein the window operations are scheduled directly by the query engine.

9. ) The tangible computer readable storage medium of claim 6, wherein the database system further executes the instructions to:

schedule window operations to process the continuous stream of data, wherein the window operations include sliding window operations and delta window operations that operate on data apart of a given time or cardinality range.

10. ) The tangible computer readable storage medium of claim 6 wherein the single long-standing query is integrated with window operations that are expressed as multilevel, multidimensional window operations in the single long-standing query.

11. ) A database system, comprising:

a database; and

a query engine in communication with the database, wherein the query engine receives a continuous stream of data, schedules window functions to process the continuous stream of data, and processes the continuous stream of data with the window functions in a single long-standing query.

12. ) The database system of claim 11, wherein a multilevel multidimensional ETL (Extracting, Transforming and Loading) mechanism uses real-time stream processing directly by the query engine.

13. ) The database system of claim 11, wherein the window functions are specified as scalar user defined functions (UDFs) that appear in a list of a query to the database.

14. ) The database system of claim 11, wherein the query engine processes multiple delta functions and sliding window functions that appear in a same query to the database.

15. ) The database system of claim 11, wherein the continuous stream of data has multiple input stream elements, and each of the window functions is called multiple times, one time for each of the multiple input stream elements.