Stream processing by a query engine

• US 8,661,014 B2
• Filed: 09/23/2010
• Issued: 02/25/2014
• Est. Priority Date: 09/23/2010
• Status: Active Grant

• Current Assignee: Snowflake, Inc.
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Chen, Qiming, Hsu, Meichun
• Primary Examiner(s): Corrielus, Jean M
• Assistant Examiner(s): ASPINWALL, EVAN S
• Application Number:
  US12/888,429
• Publication Number:
  US 20120078868A1
• Time in Patent Office: 1,251 Days
• Field of Search: 707/615, 707/648, 707/670, 707/682, 707/684, 707/685, 707/702, 707/703, 707/713, 707/769, 707/826, 707/706, 709/231
• US Class Current: 707/706
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

A database system executes a method that receives, at a query engine in the database system, a continuous stream of data. The query engine continuously analyzes the continuous stream of data with window functions in a single long-standing query.

Claims

1. A method executed by a computer system, comprising:

receiving, at a query engine in the computer system, a continuous stream of data;

pushing down processing of the continuous stream of data to a management layer where the query engine executes an application requesting a query of the continuous stream of data using User Defined Functions (UDFs) and performs stream processing of the continuous stream of data; and

continuously analyzing, inside the query engine, the continuous stream of data with multilevel, multidimensional window functions in a single long-standing query.

2. The method of claim 1, wherein the query engine continuously analyses the continuous stream of data as opposed to first storing the continuous stream of data on a disk and then analyzing the continuous stream of data.

3. The method of claim 1 further comprising, issuing the single long-standing query only once to continuously analyze the continuous stream of data as opposed to repeatedly issuing a query on the continuous stream of data.

4. The method of claim 1 further comprising, building the multilevel, multidimensional window functions for the query engine to process the continuous stream of data, wherein the multilevel, multidimensional window functions keep a history of a data state in a window of time.

5. A tangible, non-transitory computer readable storage medium storing instructions that, when executed by a query engine in a computer system, cause the query engine to:

receive, at the query engine, a continuous stream of data;

issue, inside the query engine, a single long-standing query to process the continuous stream of data;

process, inside the query engine, the continuous stream of data with multilevel, multidimensional window functions in the single long-standing query; and

schedule the multilevel, multidimensional window functions to process the continuous stream of data, wherein the multilevel, multidimensional window functions include sliding window operations and delta window operations that operate on data apart of a given time or cardinality range.

6. The tangible, non-transitory computer readable storage medium of claim 5, wherein the query engine analyzes the continuous stream of data in real-time before the continuous stream of data is stored in a database.

7. The tangible, non-transitory computer readable storage medium of claim 5, wherein query engine further executes the instructions to:

schedule the multilevel, multidimensional window functions to process the continuous stream of data, wherein the multilevel, multidimensional window functions are scheduled directly by the query engine.

8. A computer system, comprising:

a database; and

a query engine in communication with the database, wherein the query engine receives a continuous stream of data, schedules multilevel, multidimensional window functions to process the continuous stream of data, and processes the continuous stream of data with the multilevel, multidimensional window functions in a single long-standing query;

wherein the multilevel, multidimensional window functions are specified as scalar user defined functions (UDFs) that appear in a list of a query to the database.

9. The computer system of claim 8, wherein a multilevel multidimensional ETL (Extracting, Transforming and Loading) mechanism uses real-time stream processing directly by the query engine.

10. The computer system of claim 8, wherein the multilevel, multidimensional window functions comprise delta functions and sliding window functions, and wherein the query engine processes multiple delta functions and sliding window functions that appear in a same query to the database.

11. The computer system of claim 8, wherein the continuous stream of data has multiple input stream elements, and each of the multilevel, multidimensional window functions is called multiple times, one time for each of the multiple input stream elements.