Method for distributed RDSMS

• US 8,805,819 B1
• Filed: 01/29/2013
• Issued: 08/12/2014
• Est. Priority Date: 03/15/2003
• Status: Expired due to Term

• Current Assignee: Guavus, Inc. (Thales SA)
• Original Assignee: SQLstream, Inc. (Thales SA)
• Inventors: Black, Damian
• Primary Examiner(s): Ly, Anh
• Application Number:
  US13/753,385
• Time in Patent Office: 560 Days
• Field of Search: 707/713, 707/769, 707/770, 707/802, 707/692, 707/773, 707/805, 707/674
• US Class Current: 707/713
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/83 Querying
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: Y10S 707/922 Communications
• CPC Class Codes: Y10S 707/954 Relational
• CPC Class Codes: Y10S 707/959 Network
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

Methods for processing streaming data records are provided that include facilities for defining and naming multiple input and output data streams using relational processing definitions and abstractions similar to those found in relational database management systems (RDBMS) and embodied as a set of communicating stream processing nodes. The result is a Distributed Data Stream Management System (DDSMS) which is also a kind of Relational Data Stream Management System (RDSMS). The former comprises a set of Relational Data Stream Management System (RDSMS) nodes, whereas the latter might contain only a single RDSMS node.

Claims

1. A stream processing configuration service for a plurality of relational data stream processing programs executing on two or more data processing computer nodes consuming and producing relational data streams that hold and maintain information to record a configuration history, said stream processing configuration service comprising:

a processor configured to communicate with said plurality of relational data stream processing programs executing on said two or more data processing computer nodes in order to maintain consistency of configuration of said relational data stream processing programs with a current configuration of said data stream processing programs;

wherein said current configuration includes at least one configuration selected from the group consisting of;

(a) said relational data stream processing programs including information regarding execution statuses of stream processing queries currently executing on said relational data stream processing programs,

(b) said stream processing queries, including current definitions of said stream processing queries,

(c) said relational data streams, including current definitions of said relational data streams and meta data,

(d) current producers of said relational data streams, and

(e) current consumers of said relational data streams, including views of said current consumers of said relational data streams;

wherein said relational data stream processing programs execute on said two or more data processing computer nodes.

2. The stream processing configuration service of claim 1 wherein at least one of said stream processing queries are expressed in an extended subset of the SQL language.

3. A method of remotely reconfiguring at least one of a plurality of relational data stream processing programs executing on two or more data processing computer nodes and communicating with one another through data communication interfaces, said method comprising:

(a) providing a configuration interface to at least one of said relational data stream processing programs executing on said two or more data processing computer nodes;

(b) providing a control interface to at least one of said relational data stream processing programs executing on said two or more data processing computer nodes;

(c) suspending execution of incoming records by at least one of said relational data stream processing programs on said two or more data processing computer nodes via said control interface and buffering up said incoming records within each said at least one of said relational data stream processing programs in which execution has been suspended;

(d) reconfiguring, through said configuration interface, at least one of said at least one of said relational data stream processing programs in which execution has been suspended;

(e) resuming execution on said two or more data processing computer nodes, via said control interface, of at least one of said relational data stream processing programs in which execution has been suspended; and

(f) reading and processing buffered up incoming records of each of said relational data stream processing programs in which execution has been resumed before reading and processing further incoming streams of data records, wherein said suspending execution and resuming execution as performed permit configuration states of said relational data stream processing programs to be modified without any loss of data from said incoming records;

wherein said relational data stream processing programs execute on said two or more data processing computer nodes.

4. A distributed data stream management system comprising:

a plurality of relational data stream processing computer nodes distributed and configured for processing streams of data records according to relational data stream processing operations without a need to include a persistent relational database element to process said streams of data records; and

a central configuration server operably connectable to said relational stream processing computer nodes to dynamically extend processing capabilities of said relational data stream processing computer nodes;

wherein said plurality of relational data stream processing computer nodes operate in a functionally equivalent manner as if a single relational data stream processing computer node processed said streams of data records according to said relational data stream processing operations;

wherein a relational data stream processing computer node of said plurality of relational data stream processing computer nodes operates on a stream of data records inputted to said stream processing computer node using said relational data stream processing operations, generates at least one new stream of data records, and outputs said at least one new stream of data records; and

wherein records in said streams of data records are provided with monotonically increasing or decreasing keys use to synchronize said relational data stream processing operations.

5. The system of claim 4, wherein said plurality of relational data stream processing computer nodes and said central configuration server operate in the cloud.

6. The system of claim 4, wherein said plurality of relational data stream processing computer nodes process only said streams of data records, rather than managing persistent relational database tables.

7. The system of claim 4, wherein said system is a single system, and said plurality of relational data stream processing computer nodes are managed as said single system to provide an infrastructure solution for processing said streams of data records over a computer network with multiple sources of and destinations for said streams of data records.

8. The system of claim 4, wherein said monotonically increasing or decreasing keys comprise time stamps.

9. The system of claim 4, wherein at least one of said plurality of relational data stream processing computer nodes is configurable to process data records in said streams of data records within a predefined window, said predefined window being predefined to select a contiguous subset of said data records in said streams of data records, and said monotonically increasing or decreasing keys associated with said data records in said streams of data records.

10. The system of claim 9, wherein said predefined window comprises a predefined sliding window, wherein said at least one of said plurality of relational data stream processing computer nodes slides said predefined sliding window over said data records of said streams of data records to iteratively process said data records within said predefined sliding window over successive operations, such that said data records with said predefined sliding window are different subsets of said streams of data records over each successive operation.

11. The system of claim 9, wherein said predefined window comprises a paged window, wherein said at least one of said plurality of relational data stream processing computer nodes slides said predefined paged window over said data records of said streams of data records to iteratively process said data records within said predefined paged window over successive operations, wherein data records occurring in said predefined paged window are different subsets of said streams of data records over each successive operation, and wherein each data record of said streams of data records occurs in said predefined paged window in only one of said successive operations.

12. The system of claim 4, wherein said central configuration server comprises a stream processing configuration service configured to communicate with said plurality of relational data stream processing computer nodes in order to maintain consistency of configuration of relational data stream processing programs operating on said plurality of relational data stream processing computer nodes with a current configuration of said central configuration server.

13. The system of claim 12, wherein said configuration of relational data stream processing programs includes at least a current configuration, of at least one of the groups consisting of:

information regarding execution statuses of currently executing stream processing queries of said relational data stream processing programs;

said stream processing queries including current definitions of said stream processing queries;

said streams of data records including current definitions and other meta data characterizing said streams of data records;

current producers of said streams of data records; and

current consumers of said streams of data records.

14. The system of claim 4, wherein said system is configurable so that each user of said system can define a customized user view by creating customized definitions of stream queries to be processed as said relational stream processing operations.

15. The system of claim 4, wherein said relational stream processing operations comprise relational stream definitions and interdependencies, and wherein said central configuration server is configured to globally optimize said system by analyzing said relational stream definitions and interdependencies and rewriting said relational stream definitions with alternative, logically equivalent, definitions that are more efficiently executable.

16. The system of claim 15, wherein said alternative, logically equivalent definitions include embedded directives used in execution of said relational stream definitions and for distributing said relational stream definitions to predetermined ones of said plurality of relational data stream processing computer nodes.

17. The system of claim 15, wherein said globally optimize function breaks up said relational stream definitions into portions and allocates said portions to predefined ones of said plurality of relational data stream processing computer nodes.