Method for Distributed RDSMS

• US 20090094195A1
• Filed: 12/11/2008
• Published: 04/09/2009
• Est. Priority Date: 03/15/2003
• Status: Active Grant

• Current Assignee: Guavus, Inc. (Thales SA)
• Original Assignee: SQLstream, Inc. (Thales SA)
• Inventors: Black, Damian
• Granted Patent: US 8,078,609 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/83 Querying
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: Y10S 707/922 Communications
• CPC Class Codes: Y10S 707/954 Relational
• CPC Class Codes: Y10S 707/959 Network
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A method and potential embodiment for processing streaming data records is described which provides facilities for defining and naming multiple input and output data streams using relational processing definitions and abstractions similar to those found in relational database management systems (RDBMS) and embodied as a set of communicating stream processing nodes. The result is a Distributed Data Stream Management System (DDSMS) which is also a kind of Relational Data Stream Management System (RDSMS). The former comprises a set of Relational Data Stream Management System (RDSMS) nodes, whereas the latter might contain only a single RDSMS node.

The invention differs from other documented RDSMS systems in its unique combination of the method, means, capabilities, interfaces to and potential embodiments of its constituent elements, which work together to provide a seamless and complete DDSMS, and in the optimization strategies and capabilities enabled therein, and in the method, means and embodiments of its stream processing nodes. It also has novel means of data recovery both from local buffered data and through recursive recovery.

A number of novel applications of DDSMS are also described which provide a new way of existing IT problems, and at the same time afford compelling business value and offer compelling technical advantages.

The invention has resulted from the inventors extensive real-world experience gained in developing stream-processing solutions and software infrastructure in the Telecommunication fields over the last six years, along with over 20 years working various software infrastructure technologies and businesses.

Claims

1-18. -18. (canceled)

19. A method of reconfiguring a plurality of data stream processing programs executing stream processing queries processing streams of data records on at least one data processing device and communicating with one another over data communications links, said method comprising:

(a) providing a plurality of said data stream processing programs executing on said at least one data processing device;

(b) processing said streams of data records within a subset of said plurality of said data stream processing programs according to a set of operations including relational stream processing operations;

(c) communicating with at least one external data producer/publisher to send input data streams including sequences of said data records that are then transmitted to input interfaces of an input subset of said plurality of data stream processing programs for processing;

(d) communicating with at least one external data consumer/subscriber to consume output data streams including sequences of said data records output from output interfaces of an output subset of said plurality of said data stream processing programs;

(e) responding to a trigger event that signifies a need to re-optimize said stream processing queries; and

(f) utilizing data including a status history including at least a current status of operating statuses of at least one of the statuses selected from the group consisting of;

status of said data communications links, status of said data stream processing programs, and status of currently executing queries; and

utilizing definitions of said streams in order to generate reconfiguration instructions that create a modified configuration of said data stream processing programs, said reconfiguration instructions including configuration changes to stream processing operations and including data stream processing program control operations, said configuration changes including replacing at least one query with a new formulation of said query to be executed upon at least one currently operational data stream processing program so as to achieve a system-wide input-output processing that functions mathematically equivalently to function of the system before said trigger event, with respect to each of said external data producers/publishers and each of said external data consumers/subscribers; and

,

issuing said reconfiguration instructions to said data stream processing programs determined as needed to put into effect said new configuration.

20. The method of claim 19, wherein no data records are lost during said reconfiguration.

21. The method of claim 19, wherein said new formulation of said query includes creating sub-queries that share at least one sub-stream of data records sending said data records to at least two queries within said new formulation thereby avoiding unnecessary recomputation of said sub-stream.

22. The method of claim 19, wherein said new formulation of said query includes creating sub-queries that through said reconfiguring divide at least one stream of data records into at least two streams of data records processed by sub-queries executing on distinct data stream processing programs in order to perform load balancing/sharing wherein stream processing work can be shared across additional data stream processing programs.

23. The method of claim 19, wherein said new formulation of said query includes at least one multi-cast transmission operation to avoid a repeated point to point transmission of data records from at least one data stream processing program to at least two destinations.

24. The method of claim 19, wherein said new formulation of said query includes holding and maintaining a current index of a subset of currently executing stream processing queries in order to avoid sending data records to irrelevant queries that would otherwise perform wasteful operations that would not contribute to results of computations of said irrelevant queries.

25. The method of claim 19, further comprising collecting a history of metrics including current metrics on at least one member of the group consisting of:

operational and execution statuses of at least one of said data stream processing programs,

at least one of said data communications links,

at least one of said streams; and

comparing said history metrics against optimization policy goals whereby sufficient divergence of said history of metrics from said optimization policy goals generates said trigger event;

whereby said modified configuration reduces divergence of said history of metrics from said optimization policy goals.

26. The method of claim 25, wherein at least one of said metrics is collected and processed by stream processing queries executing upon one or more of said data stream processing programs.

27. The method of claim The method of claim 25, wherein at least one of said metrics collected is a quality of service metric, and wherein at least one of said optimization policy goals is a quality compliance threshold, and wherein when said quality compliance threshold has been exceeded, said modified configuration comprises a remedial reconfiguration of said data stream processing programs to improve quality whereby sets of data stream processing programs adaptively reconfigure to maintain said quality compliance threshold.

28. The method of claim 25, wherein said quality compliance threshold comprises at least one measure selected from the group consisting of:

measures of desired performance for a set of said streams, measures of desired system-wide performance covering all of said streams, and composite measures comprising sets of desired performance measures such that each of said measures defines a predefined measure of desired quality of service for a set of streams.

29. The method of claim 25, further comprising generating said trigger event upon the occurrence of at least one of:

passing of a scheduled time interval, reaching a pre-specified clock time, receiving an external trigger instruction, crossing a specified threshold of available resources, detecting a change in availability of data stream processing programs, detecting a change in availability of data processing devices, and detecting a change in availability of said data communication links whereby said data stream processing programs reconfigure and adapt