Real-time mining and reduction of streamed data

• US 8,478,889 B2
• Filed: 02/18/2009
• Issued: 07/02/2013
• Est. Priority Date: 09/30/2005
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bestgen, Robert Joseph, Carlson, David Glenn, Kathmann, Kevin James
• Primary Examiner(s): Dharia, Rupal
• Assistant Examiner(s): Nguyen, Van Kim T
• Application Number:
  US12/372,850
• Publication Number:
  US 20090150560A1
• Time in Patent Office: 1,595 Days
• Field of Search: 709/231, 709/232, 707/718, 707/721, 707/769
• US Class Current: 709/231
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

A stream data node receives real-time streamed data from one or more input devices, dynamically filters the streamed data to reduce the streamed data, and delivers the reduced data when requested. By providing real-time filtering of the data, the amount of data that must be stored in a database may be substantially reduced. The stream data node can perform aggregation functions, group functions, and select functions, thereby also significantly reducing the amount of data that must be stored in a database. The stream data node may also be part of a query execution data structure, where it delivers its data when requested by another node in the query execution data structure.

Claims

1. An apparatus comprising:

at least one processor;

a memory coupled to the at least one processor; and

a stream data node residing in the memory that receives streamed data from at least one data source, dynamically reduces the streamed data using a filter mechanism that specifies data to exclude from the streamed data according to at least one filter criterion as the streamed data is received before storing the streamed data in a database, and delivers the reduced streamed data when an external request is received, wherein the stream data node is part of a query execution data structure representing a query execution plan for querying data stored in database tables, wherein the query execution data structure includes at least one join node, and wherein the stream data node delivers the reduced streamed data when requested by a different node in the query execution data structure.

2. The apparatus of claim 1 wherein the at least one filter criterion specifies an aggregation function.

3. The apparatus of claim 1 wherein the at least one filter criterion specifies a group function.

4. The apparatus of claim 1 wherein the at least one filter criterion specifies a select function.

5. A computer-implemented method for processing streamed data from at least one data source, the method comprising the steps of:

(A) providing at least one processor;

(B) providing a computer readable memory coupled to the at least one processor;

(C) a stream data node receiving the streamed data from the at least one data source, wherein the stream data node is part of a query execution data structure representing a query execution plan for querying data stored in database tables, wherein the query execution data structure includes at least one join node;

(D) the stream data node dynamically reducing the streamed data using a filter mechanism that specifies data to exclude from the streamed data according to at least one filter criterion as the streamed data is received before storing the streamed data in a database in the memory; and

(E) the stream data node delivering the reduced streamed data from the database in the memory when requested by a different node in the query execution data structure.

6. The method of claim 5 further comprising the step of buffering in the memory the streamed data received in step (C).

7. The method of claim 5 wherein the at least one filter criterion specifies an aggregation function.

8. The method of claim 5 wherein the at least one filter criterion specifies a group function.

9. The method of claim 5 wherein the at least one filter criterion specifies a select function.

10. The method of claim 5 wherein step (E) is performed when the external request is made by a node in a query execution data structure.

11. A computer-readable program product comprising software residing on non-transitory media, the software comprising:

a stream data node that receives streamed data from at least one data source, dynamically reduces the streamed data using a filter mechanism that specifies data to exclude from the streamed data according to at least one filter criterion as the streamed data is received before storing the streamed data in a database, and delivers the reduced streamed data when an external request is received, wherein the stream data node is part of a query execution data structure representing a query execution plan for querying data stored in database tables, wherein the query execution data structure includes at least one join node, and wherein the stream data node delivers the reduced streamed data when requested by a different node in the query execution data structure.

12. The program product of claim 11 wherein the at least one filter criterion specifies an aggregation function.

13. The program product of claim 11 wherein the at least one filter criterion specifies a group function.

14. The program product of claim 11 wherein the at least one filter criterion specifies a select function.