Method for distributed RDSMS

• US 8,078,609 B2
• Filed: 12/11/2008
• Issued: 12/13/2011
• Est. Priority Date: 03/15/2003
• Status: Active Grant

• Current Assignee: Guavus, Inc. (Thales SA)
• Original Assignee: SQLstream, Inc. (Thales SA)
• Inventors: Black, Damian
• Primary Examiner(s): Fleurantin, Jean B
• Assistant Examiner(s): Ly, Anh
• Application Number:
  US12/333,230
• Publication Number:
  US 20090094195A1
• Time in Patent Office: 1,097 Days
• Field of Search: 707/713, 707/769, 707/770
• US Class Current: 707/713
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/83 Querying
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: Y10S 707/922 Communications
• CPC Class Codes: Y10S 707/954 Relational
• CPC Class Codes: Y10S 707/959 Network
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A method and potential embodiment for processing streaming data records is described which provides facilities for defining and naming multiple input and output data streams using relational processing definitions and abstractions similar to those found in relational database management systems (RDBMS) and embodied as a set of communicating stream processing nodes. The result is a Distributed Data Stream Management System (DDSMS) which is also a kind of Relational Data Stream Management System (RDSMS). The former comprises a set of Relational Data Stream Management System (RDSMS) nodes, whereas the latter might contain only a single RDSMS node. A unique combination of the method, means, capabilities, interfaces to and potential embodiments of its constituent elements, work together to provide a seamless and complete DDSMS, and in the optimization strategies and capabilities enabled therein, and in the method, means and embodiments of its stream processing nodes.

Claims

1. A method of reconfiguring a plurality of data stream processing programs executing stream processing queries processing streams of data records on at least one data processing node and communicating with one another over data communications links, said method comprising:

(a) providing a plurality of said data stream processing programs executing on said at least one data processing node;

(b) processing said streams of data records within a subset of said plurality of said data stream processing programs according to a set of operations including relational stream processing operations;

(c) communicating with at least one external data producer/publisher to send input data streams including sequences of said data records that are then transmitted to input interfaces of an input subset of said plurality of data stream processing programs for processing;

(d) communicating with at least one external data consumer/subscriber to consume output data streams including sequences of said data records output from output interfaces of an output subset of said plurality of said data stream processing programs;

(e) utilizing data including at least a current status of operating statuses of at least one of the statuses selected from the group consisting of;

status of said data communications links, status of said data stream processing programs, and status of currently executing queries; and

utilizing definitions of said streams in order to generate reconfiguration instructions that create a modified configuration of said data stream processing programs, said reconfiguration instructions including configuration changes to stream processing operations and including data stream processing program control operations, said configuration changes including replacing at least one query with a new query to be executed upon at least one currently operational data stream processing program so as to achieve a system-wide input-output processing that functions mathematically equivalently to function of the system before implementation of said new query, with respect to each of said external data producers/publishers and each of said external data consumers/subscribers; and

(f) issuing said reconfiguration instructions to said data stream processing programs determined as needed to put into effect said modified configuration.

2. The method of claim 1, wherein no data records are lost during said reconfiguration.

3. The method of claim 1, wherein said replacing with a reformulated relational expression includes creating sub-stream of data records and sending said data records to at least two queries within said new formulation thereby avoiding unnecessary recomputation of said sub-stream.

4. The method of claim 1, wherein said a reformulated relational expression includes creating separate sub-streams from a single stream generating operation that through said reconfiguring divide at least one stream of data records into at least two streams of data records processed by sub queries executing on distinct data stream processing programs in order to perform load balancing/sharing wherein stream processing work can be shared across additional data stream processing programs.

5. The method of claim 1, wherein said reformulated relational expression includes at least one multi-cast transmission operation to avoid a repeated point to point transmission of data records from at least one data stream processing program to at least two destinations.

6. The method of claim 1, wherein said reformulated relational expression includes holding and maintaining a current index of a subset of currently executing stream processing queries in order to provide efficient transmission of data records to relevant queries.

7. The method of claim 1, further comprising:

collecting metrics including current metrics on at least one member of the group consisting of;

operational and execution statuses of at least one of said data stream processing programs, at least one of said data communications links, and at least one of said streams; and

comparing said metrics against optimization policy goals whereby sufficient divergence of said metrics from said optimization policy goals initiates generation of said reconfiguration instructions;

whereby said modified configuration reduces divergence of said metrics from said optimization policy goals.

8. The method of claim 7, wherein at least one of said metrics is collected and processed by stream processing queries executing upon one or more of said data stream processing programs.

9. The method of claim 7, wherein at least one of said metrics collected is a quality of service metric, and wherein at least one of said optimization policy goals is a quality compliance threshold, and wherein when said quality compliance threshold has been exceeded, said modified configuration comprises a remedial reconfiguration of said data stream processing programs to improve quality whereby sets of data stream processing programs adaptively reconfigure to maintain said quality compliance threshold.

10. The method of claim 7, wherein said quality compliance threshold comprises at least one measure selected from the group consisting of:

measures of desired performance for a set of said streams, measures of desired system-wide performance covering all of said streams, and composite measures comprising sets of desired performance measures such that each of said measures defines a predefined measure of desired quality of service for a set of streams.

11. A computer-implemented method of reconfiguring a plurality of data stream processing programs executing stream processing queries represented as relational expressions comprising sets of relational operators that operate on streams of data records, processing said streams of data records on at least one data processing node and communicating with one another over data communications links, said method comprising:

(a) providing a plurality of said data stream processing programs executing on said at least one data processing node;

(b) processing said streams of data records within a subset of said plurality of said data stream processing programs according to a set of operations including relational stream processing operations;

(c) communicating with at least one external data producer/publisher to send input data streams including sequences of said data records that are then transmitted to input interfaces of an input subset of said plurality of data stream processing programs for processing;

(d) communicating with at least one external data consumer/subscriber to consume output data streams including sequences of said data records output from output interfaces of an output subset of said plurality of said data stream processing programs;

(e) responding to a trigger event that signifies a need to re-optimize said stream processing queries; and

(e) utilizing data including a status history including at least a current status of operating statuses of at least one of the statuses selected from the group consisting of;

status of said data communications links, status of said data stream processing programs, and status of currently executing queries; and

utilizing definitions of said streams in order to generate reconfiguration instructions that create a modified configuration of said data stream processing programs, said reconfiguration instructions including configuration changes to stream processing operations and including data stream processing program control operations, said configuration changes including replacing at least one relational expression of at least one query with a reformulated relational expression of said at least one relational expression to be executed upon at least one currently operational data stream processing program so as to achieve a system-wide input-output processing that functions mathematically equivalently to function of the system before reformulation of said reformulated relational expression, with respect to each of said external data producers/publishers and each of said external data consumers/subscribers; and

,

(f) issuing said reconfiguration instructions to said data stream processing programs determined as needed to put into effect said modified˜

configuration.