System and Method For Querying Data Streams

• US 20110173231A1
• Filed: 01/11/2010
• Published: 07/14/2011
• Est. Priority Date: 01/11/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Grandison, Tyrone Wilberforce Andre, Paraszczak, Jurij R., Harrison, Colin George, Eckman, Barbara Ann, Hartswick, Perry G., Williams, Ralph Peter, Drissi, Youssef
• Granted Patent: US 8,423,576 B2
• Time in Patent Office: Days
• US Class Current: 707/771
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

A method includes parsing input from a requestor, where the input includes at least one of a query and a command that is parsed into a structured query having an indication of at least one data stream or set of data streams and at least one action to be performed on the at least one data stream or set of data streams. The method further includes mapping the structured query into a graph of processing elements that are selected and interconnected so as to execute the structured query; instantiating the graph of processing elements and connecting and initializing the instantiated graph of processing elements with an identified at least one data stream or set of data streams to receive data there from; and outputting a result of the structured query to the requestor.

Claims

1. A method to process data, comprising:

parsing input from a requestor, the input comprising at least one of a query or a command that is parsed into a structured query comprising an indication of at least one data stream or set of data streams and at least one action to be performed on the, at least one, data stream or set of data streams;

mapping the structured query into a graph of processing elements that are selected and interconnected so as to execute the structured query;

instantiating the graph of processing elements and connecting and initializing the instantiated graph of processing elements with an identified, at least one, data stream or set of data streams to receive data there from; and

outputting a result of the structured query to the requestor.

2. The method of claim 1, further comprising monitoring operation of the instantiated graph of processing elements during operation thereof, and notifying the requestor of an occurrence of an exception condition.

3. The method of claim 1, where the requestor is one of a user or a software application.

4. The method of claim 1, where the processing elements are selected and interconnected so as to determine at least one of a longitudinal characteristic of an individual data stream;

a longitudinal characteristic of a sub-set of data streams;

a transverse characteristic of an individual data stream;

an auto-correlation of an attribute of an individual data stream or set of data streams;

a correlation between individual data streams or sets of data streams of different types; and

longitudinal and correlation attributes of time series data across one or more data streams or sets of data streams.

5. The method of claim 1, where a result of execution of the command is to at least one of merge or combine together two or more data streams, split a data stream into two or more data streams, and interleave at least two streams.

6. The method of claim 1, where at least one data stream is a data stream comprising data being generated in real time or substantially real time.

7. The method of claim 1, where at least one data stream is a data stream comprising previously generated and stored data.

8. The method of claim 1, where at least one data stream is a data stream comprising data generated from at least one of a forecast and a model.

9. The method of claim 1, where the result comprises an identifier of at least one data stream.

10. The method of claim 1, where the input from the requestor that is parsed comprises the at least one of the query or the command expressed as a natural language query or command.

11. The method of claim 1, where a particular stream is one of data output from a sensor, a computed stream, or a re-streaming of historical data, and where an output of the instantiated graph of processing elements is a sink that comprises a storage medium and a file system.

12. A data processing system, comprising:

at least one data processor coupled with at least one memory that stores computer software, where execution of the computer software by the at least one data processor results in the system performing operations that comprise,

parsing input from a requestor comprised of at least one of a user and a software application, the input comprising at least one of a query or a command that is parsed into a structured query comprising an indication of at least one data stream or set of data streams and at least one action to be performed on the at least one data stream or set of data streams;

mapping the structured query into a graph of processing elements that are selected and interconnected so as to execute the structured query;

instantiating the graph of processing elements and connecting and initializing the instantiated graph of processing elements with an identified at least one data stream or set of data streams to receive data there from; and

outputting a result of the structured query to the requestor.

13. The data processing system of claim 12, further comprising an operation of monitoring operation of the instantiated graph of processing elements during operation thereof, and notifying the requestor of an occurrence of an exception condition.

14. The data processing system of claim 12, where the processing elements are selected and interconnected so as to determine at least one of a longitudinal characteristic of an individual data stream;

a longitudinal characteristic of a sub-set of data streams;

a transverse characteristic of an individual data stream;

an auto-correlation of an attribute of an individual data stream or set of data streams;

a correlation between individual data streams or sets of data streams of different types; and

longitudinal and correlation attributes of time series data across one or more data streams or sets of data streams, and where a result of execution of the command is to at least one of merge or combine together two or more data streams, split a data stream into two or more data streams, and interleave at least two streams.

15. The data processing system of claim 12, where at least one data stream is one of a data stream comprising data being generated in real time or substantially real time, a data stream comprising previously generated and stored data, and a data stream comprising data generated from at least one of a forecast and a model.

16. The data processing system of claim 12, where the result comprises an identifier of at least one data stream.

17. The data processing system of claim 12, where the input from the requestor that is parsed comprises the at least one of the query and the command expressed as a natural language query or command.

18. The data processing system of claim 12, where a particular stream is one of data output from a sensor, a computed stream, or a re-streaming of historical data, and where an output of the instantiated graph of processing elements is a sink that comprises a storage medium and a file system.

19. A computer-readable medium that stores computer software, where execution of the computer software by at least one data processor results in operations that comprise,

parsing input from a requestor comprised of at least one of a user and a software application, the input comprising at least one of a query or a command that is parsed into a structured query comprising an indication of at least one data stream or set of data streams and at least one action to be performed on the at least one data stream or set of data streams;

mapping the structured query into a graph of processing elements that are selected and interconnected so as to execute the structured query;

instantiating the graph of processing elements and connecting and initializing the instantiated graph of processing elements with an identified at least one data stream or set of data streams to receive data there from; and

outputting a result of the structured query to the requestor.

20. The computer-readable medium of claim 19, further comprising an operation of monitoring operation of the instantiated graph of processing elements during operation thereof, and notifying the requestor of an occurrence of an exception condition.

21. The computer-readable medium of claim 19, where the processing elements are selected and interconnected so as to determine at least one of a longitudinal characteristic of an individual data stream;

a longitudinal characteristic of a sub-set of data streams;

a transverse characteristic of an individual data stream;

an auto-correlation of an attribute of an individual data stream or set of data streams;

a correlation between individual data streams or sets of data streams of different types; and

longitudinal and correlation attributes of time series data across one or more data streams or sets of data streams, and where a result of execution of the command is to at least one of merge or combine together two or more data streams, split a data stream into two or more data streams, and interleave at least two streams.

22. The computer-readable medium of claim 19, where at least one data stream is one of a data stream comprising data being generated in real time or substantially real time, a data stream comprising previously generated and stored data, and a data stream comprising data generated from at least one of a forecast and a model.

23. The computer-readable medium of claim 19, where the result comprises an identifier of at least one data stream.

24. The computer-readable medium of claim 19, where the input from the requestor that is parsed comprises the at least one of the query and the command expressed as a natural language query or command.

25. The computer-readable medium of claim 19, where a particular stream is one of data output from a sensor, a computed stream, or a re-streaming of historical data, and where an output of the instantiated graph of processing elements is a sink that comprises a storage medium and a file system.