Method for distributed RDSMS

• US 7,010,538 B1
• Filed: 03/15/2003
• Issued: 03/07/2006
• Est. Priority Date: 03/15/2003
• Status: Active Grant

• Current Assignee: Guavus, Inc. (Thales SA)
• Original Assignee: Damian Black
• Inventors: Black, Damian
• Primary Examiner(s): Corrielus, Jean M.
• Assistant Examiner(s): Ly, Anh
• Application Number:
  US10/390,091
• Time in Patent Office: 1,088 Days
• Field of Search: 707100-1041
• US Class Current: 707/636
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/83 Querying
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: Y10S 707/922 Communications
• CPC Class Codes: Y10S 707/954 Relational
• CPC Class Codes: Y10S 707/959 Network
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A method and potential embodiment for processing streaming data records is described which provides facilities for defining and naming multiple input and output data streams (along with their related transformation, filtering, aggregation and other processing operations) using relational processing definitions, abstractions and techniques similar to those found in relational database management systems (RDBMS) and which is embodied as a set of communicating stream processing nodes. Whereas RDBMS process queries on a centralized processing node against historical stored data records, DDSMS process queries on distributed processing nodes against future flowing data records. The result is a Distributed Data Stream Management System (DDSMS) comprising a set of relational stream processing nodes that work and communicate together in concert to perform useful distributed data processing for a variety of applications such as service monitoring, alerting, message brokering and other applications.

Claims

1. A method of managing and processing streams of data records over a data communication network means linking a plurality of data processing computing means comprising:

(a) providing a plurality of stream processing node means for consuming and generating streams of data records executing on said data processing computing means,

(b) providing a central configuration store means for holding configuration data describing the names of said streams, and describing relational stream processing operations and other configuration and definition data of said streams, and for holding configuration data of said stream processing node means, said central configuration store means executing on said data processing computing means,

(c) configuring said stream processing node means using a user-interface means to manage the configuration data in said central configuration store means, which in turn communicates over said data communication network means with a collection of said stream processing node means to effect configuration changes needed therein to maintain consistency of configuration data of said stream processing node means with respect to the configuration data held in said central configuration store means,

(d) generating with a translation means said relational stream processing operations from relational stream processing definitions input through said user-interface means and then storing said relational stream processing operations in said central configuration store means, after which said configuration store means communicates with said stream processing node means to effect configuration changes therein,

(e) communicating with external data generating means to create input streams for processing by said stream processing node means,

(f) communicating with external data processing means to consume output streams of said stream processing node means,

(g) transmitting using said data communication network means data records in the form of streams comprising sequences of data records each of which comprises a set of named data fields, said transmitting being from some of said stream processing node means to others of said stream processing node means in order to be further processed according to the definitions of the configuration data held in said central configuration store means,

(h) processing streams transmitted to the input of each of said stream processing node means and creating output streams from said stream processing node means, said processing being performed within each of said stream processing node means according to a set of relational stream processing operations by a relational stream engine means, and where said relational stream engine means has a plug-in management means for extending its stream processing capabilities,

(i) controlling said stream processing node means through a control interface means of each stream processing node means in order to control the execution of said stream processing node means,

(j) configuring said stream processing node means through a configuration interface means of each stream processing node means, and

(k) allowing each of said stream processing node means to interoperate with one another through said control interface means in order to control the transmission of records between said stream processing node means, whereby data records of said external data generating means are processed generating data records for consumption by said external data processing means according to the relational stream processing operations stored in said central configuration store means and using a plurality of said stream processing nodes means to perform said processing.

2. The method of claim 1 wherein said relational stream processing definitions are written in the sql 2003 language, said language being the relational data manipulation language standardized by the American National Standards Institute.

3. The method of claim 1 further performing global optimization according to a set of optimization criteria stored in said central configuration store means, said global optimization comprising analyzing the configuration data of said central configuration store means and generating and storing new configuration data in said central configuration store means, and consequently changing the configuration of said stream processing node means while preserving the input-output behavior of the complete system with respect to the data of external data generating means and the data of external data processing means, in order to reduce resources required for execution by said stream processing node means whereby the processing performance of said stream processing node means when operating as a complete system is optimized.

4. The method of claim 1 further performing local optimization within stream processing node means, said local optimization comprising analyzing the configuration data of said stream processing node means and modifying the execution plan of said stream processing node means while preserving the input-output behavior of said stream processing node means with respect to the input data and the output data of said stream processing node means, in order to reduce computational resources required for executing said execution plan.

5. The method of claim 1, further providing recovery processing where stream processing node means communicate over said data communication network means to other relevant stream processing node means to request retransmission of data records in order to recover from detected failures.

6. The method of claim 1 wherein all of said stream processing node means execute on a single data processing computing means.

7. The method of claim 1 wherein said stream processing node means store streams in a recovery buffer means for possible retransmission.

8. The method of claim 7 wherein said recovery buffers are stored in persistent store means.

9. The method of claim 1 wherein the relational stream processing of records is performed in real-time according to timing constraints stored in said central configuration store means.

10. The method of claim 1 wherein communicating with external data generating means results in generating streams of records each of which are automatically augmented with a timestamp field indicating their time of creation.

11. The method of claim 10 wherein the processing of records is performed in real-time, the timing of said processing being governed by timing constraints stored in said central configuration store means.

12. The method of claim 1 wherein said external data generating means generate records in the format produced by some external software application components and said external data processing means consume records output by stream processing node means, where said application components are executing on some computing means, and wherein said method thereby translates records from said format to the required input formats of other application components, whereby benefits are afforded of simple and efficient data transformation and insulation of said application components from changes to one another.

13. The method of claim 1 wherein said external data generating means generate records representing messages produced by external software application components and said external data processing means consume said messages, where said application components are executing on some computing means, and wherein said method thereby brokers messages between said application components, whereby benefits are afforded of superior message flexibility, superior message throughput and superior system manageability relative to those of conventional message broker solutions, and also the benefit is afforded of easy real-time matching of messages with the same fields and values.

14. The method of claim 1 wherein said external data generating means generate records representing streaming market data, said records are processed using a plurality of stream processing nodes and said external data processing means comprise a set of consumers of streaming preprocessed market data records, and wherein said method thereby distributes said streaming preprocessed market data records to said set of consumers whereby benefits are afforded of superior preprocessing flexibility, superior system throughput and superior system manageability relative to conventional solutions based on message brokers, and also the benefit is afforded of easy real-time matching of records with common market instrument identity fields.

15. The method of claim 1 wherein said external data generating means generate records representing streaming sensor data, said records are processed using a plurality of stream processing node means to generate output records consumed by a set of consumers of streaming preprocessed sensor data records, whereby benefits are afforded of superior preprocessing flexibility, superior throughput and superior system manageability relative to conventional solutions based on message brokers, and also the benefit is afforded of easy real-time matching of records with common sensor identity fields within a specified time window.

16. The method of claim 1 wherein said external data generating means generate records representing streaming location data records for mobile devices and said external data processing means comprise a set of consumers of streaming preprocessed location based data records, and wherein said method thereby distributes said streaming preprocessed location data records to said set of consumers whereby benefits are afforded of superior preprocessing flexibility, superior throughput, and superior system manageability relative to conventional solutions based on message brokers, and also the benefit is afforded of easy real-time matching of records with the same location fields within a specified time window.

17. The method of claim 1 wherein said external data generating means generate records representing streaming sources of service component data records and said external data processing means comprise a plurality of consumers of streaming preprocessed service detail records, and wherein said method thereby merges, aggregates, and otherwise assembles complete service detail records from said streaming sources whereby benefits are afforded of superior preprocessing flexibility, superior throughput and superior system manageability relative to those of conventional service data mediation solutions.

18. The method of claim 1 wherein said external data generating means generate records representing streaming sources of component data records and said external data processing means comprise a set of consumers of streaming preprocessed data records, and wherein said method thereby merges, aggregates, and otherwise assembles complete data records from said streaming sources whereby benefits are afforded of superior preprocessing flexibility, superior throughput and superior system manageability relative to those of conventional extract, translate and load systems.