Apparatus and method for real-time mining and reduction of streamed data

• US 20070078802A1
• Filed: 09/30/2005
• Published: 04/05/2007
• Est. Priority Date: 09/30/2005
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Bestgen, Robert, Carlson, David, Kathmann, Kevin
• Granted Patent: US 7,512,700 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

A stream data node receives real-time streamed data from one or more input devices, dynamically filters the streamed data to reduce the streamed data, and delivers the reduced data when requested. By providing real-time filtering of the data, the amount of data that must be stored in a database may be substantially reduced. The stream data node can perform aggregation functions, group functions, and select functions, thereby also significantly reducing the amount of data that must be stored in a database. The stream data node may also be part of a query execution data structure, where it delivers its data when requested by another node in the query execution data structure.

Claims

1. An apparatus comprising:

at least one processor;

a memory coupled to the at least one processor; and

a stream data node residing in the memory that receives streamed data from at least one data source, dynamically reduces the streamed data, and delivers the reduced streamed data when an external request is received.

2. The apparatus of claim 1 wherein the stream data node comprises a filter mechanism that dynamically reduces the streamed data according to at least one filter criterion.

3. The apparatus of claim 2 wherein the at least one filter criterion specifies an aggregation function.

4. The apparatus of claim 2 wherein the at least one filter criterion specifies a group function.

5. The apparatus of claim 2 wherein the at least one filter criterion specifies a select function.

6. The apparatus of claim 1 wherein the stream data node is part of a query execution data structure, and wherein the stream data node delivers the reduced streamed data when requested by a different node in the query execution data structure.

7. A computer-implemented method for processing streamed data from at least one data source, the method comprising the steps of:

(A) receiving the streamed data from the at least one data source;

(B) dynamically reducing the streamed data as the streamed data is received before storing the streamed data in a database; and

(C) delivering the reduced streamed data when an external request is received.

8. The method of claim 7 further comprising the step of buffering the streamed data received in step (A).

9. The method of claim 7 wherein step (B) comprises the step of filtering the streamed data according to at least one filter criterion.

10. The method of claim 9 wherein the at least one filter criterion specifies an aggregation function.

11. The method of claim 9 wherein the at least one filter criterion specifies a group function.

12. The method of claim 9 wherein the at least one filter criterion specifies a select function.

13. The method of claim 7 wherein step (C) is performed when the external request is made by a node in a query execution data structure.

14. A computer-readable program product comprising:

(A) a stream data node that receives streamed data from at least one data source, dynamically reduces the streamed data, and delivers the reduced streamed data when an external request is received; and

(B) computer-readable signal bearing media bearing the stream data node.

15. The program product of claim 14 wherein the computer-readable signal bearing media comprises recordable media.

16. The program product of claim 14 wherein the computer-readable signal bearing media comprises transmission media.

17. The program product of claim 14 wherein the stream data node comprises a filter mechanism that dynamically reduces the streamed data according to at least one filter criterion.

18. The program product of claim 17 wherein the at least one filter criterion specifies an aggregation function.

19. The program product of claim 17 wherein the at least one filter criterion specifies a group function.

20. The program product of claim 17 wherein the at least one filter criterion specifies a select function.

21. The program product of claim 14 wherein the stream data node is part of a query execution data structure, and wherein the stream data node delivers the reduced streamed data when requested by a different node in the query execution data structure.

22. A networked computer system comprising:

(A) at least one data source that provides a data output comprising streamed data; and

(B) a computer system coupled via at least one network connection to the at least one data source, the computer system comprising;

at least one processor;

a memory coupled to the at least one processor; and

a stream data node residing in the memory that receives the streamed data from the at least one data source, dynamically reduces the streamed data, and delivers the reduced streamed data when an external request is received.