REAL-TIME MINING AND REDUCTION OF STREAMED DATA

• US 20090150560A1
• Filed: 02/18/2009
• Published: 06/11/2009
• Est. Priority Date: 09/30/2005
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bestgen, Robert Joseph, Kathmann, Kevin James, Carlson, David Glenn
• Granted Patent: US 8,478,889 B2
• Time in Patent Office: Days
• US Class Current: 709/231
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

A stream data node receives real-time streamed data from one or more input devices, dynamically filters the streamed data to reduce the streamed data, and delivers the reduced data when requested. By providing real-time filtering of the data, the amount of data that must be stored in a database may be substantially reduced. The stream data node can perform aggregation functions, group functions, and select functions, thereby also significantly reducing the amount of data that must be stored in a database. The stream data node may also be part of a query execution data structure, where it delivers its data when requested by another node in the query execution data structure.

Claims

1. An apparatus comprising:

at least one processor;

a memory coupled to the at least one processor; and

a stream data node residing in the memory that receives streamed data from at least one data source, dynamically reduces the streamed data using a filter mechanism that specifies data to exclude from the streamed data according to at least one filter criterion as the streamed data is received before storing the streamed data in a database, and delivers the reduced streamed data when an external request is received.

3. The apparatus of claim 1 wherein the at least one filter criterion specifies an aggregation function.

4. The apparatus of claim 1 wherein the at least one filter criterion specifies a group function.

5. The apparatus of claim 1 wherein the at least one filter criterion specifies a select function.

6. The apparatus of claim 1 wherein the stream data node is part of a query execution data structure, and wherein the stream data node delivers the reduced streamed data when requested by a different node in the query execution data structure.

2. (canceled)

7. A computer-implemented method for processing streamed data from at least one data source, the method comprising the steps of:

(A) providing at least one processor;

(B) providing a computer readable memory coupled to the at least one processor;

(C) receiving the streamed data from the at least one data source;

(D) dynamically reducing the streamed data using a filter mechanism that specifies data to exclude from the streamed data according to at least one filter criterion as the streamed data is received before storing the streamed data in a database in the memory; and

(E) delivering the reduced streamed data from the database in the memory when an external request is received.

8. The method of claim 7 further comprising the step of buffering in the memory the streamed data received in step (C).

10. The method of claim 7 wherein the at least one filter criterion specifies an aggregation function.

11. The method of claim 7 wherein the at least one filter criterion specifies a group function.

12. The method of claim 7 wherein the at least one filter criterion specifies a select function.

13. The method of claim 7 wherein step (E) is performed when the external request is made by a node in a query execution data structure.

9. (canceled)

14. A computer-readable program product comprising:

(A) a stream data node that receives streamed data from at least one data source, dynamically reduces the streamed data using a filter mechanism that specifies data to exclude from the streamed data according to at least one filter criterion as the streamed data is received before storing the streamed data in a database, and delivers the reduced streamed data when an external request is received; and

(B) recordable media bearing the stream data node.

16. The program product of claim 14 wherein the at least one filter criterion specifies an aggregation function.

17. The program product of claim 14 wherein the at least one filter criterion specifies a group function.

18. The program product of claim 14 wherein the at least one filter criterion specifies a select function.

19. The program product of claim 14 wherein the stream data node is part of a query execution data structure, and wherein the stream data node delivers the reduced streamed data when requested by a different node in the query execution data structure.

15. (canceled)