Method for distributed RDSMS

• US 8,412,733 B1
• Filed: 11/07/2011
• Issued: 04/02/2013
• Est. Priority Date: 03/15/2003
• Status: Active Grant

• Current Assignee: Guavus, Inc. (Thales SA)
• Original Assignee: SQL Stream Incorporated
• Inventors: Black, Damian
• Primary Examiner(s): Ly, Anh
• Application Number:
  US13/290,886
• Time in Patent Office: 512 Days
• Field of Search: 707/664, 707/692, 707/769, 707/773, 707/805, 707/802, 707/674, 709/237, 709/231, 709/230
• US Class Current: 707/769
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/83 Querying
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: Y10S 707/922 Communications
• CPC Class Codes: Y10S 707/954 Relational
• CPC Class Codes: Y10S 707/959 Network
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

Methods for processing streaming data records are provided that include facilities for defining and naming multiple input and output data streams using relational processing definitions and abstractions similar to those found in relational database management systems (RDBMS) and embodied as a set of communicating stream processing nodes. The result is a Distributed Data Stream Management System (DDSMS) which is also a kind of Relational Data Stream Management System (RDSMS). The former comprises a set of Relational Data Stream Management System (RDSMS) nodes, whereas the latter might contain only a single RDSMS node.

Claims

1. A computer-implemented method of retransmitting data when processing streams of data records using a plurality of relational data stream processing programs executing on at least one data processing device and communicating with one another through data communication interfaces to form a network of a plurality of relational data stream processing programs, said method comprising:

providing said plurality of relational data stream processing programs executing on said at least one data processing device for processing said streams of data records;

processing said streams of data records within two or more of said plurality of said relational data stream processing programs according to stream processing operations including relational stream processing operations;

detecting a need to retransmit at least one missing data record of said streams of data records; and

recovering said at least one missing data record by retrieving said at least one missing data record from a buffer of a relational data stream processing program upstream of a location where said need to retransmit has been detected, wherein said relational data stream processing program is the relational data stream processing program that originally sent said at least one missing data record to said location where said need to retransmit has been detected said relational data stream processing program upstream of the location, and said relational data stream processing program upstream of the location resending said at least one missing data record to at least one other of said relational data stream processing programs that needs said retransmitting.

2. The computer-implemented method of claim 1, further comprising:

utilizing data records currently held by said at least one of said relational data stream processing programs and by also utilizing definitions of queries that produce said streams of data records in order to create data recovery instructions comprising specific stream processing operations and relational data stream processing program control operations required to perform data reconstitution and retransmission; and

issuing said data recovery instructions to said at least one of said relational data stream processing programs determined as needed to perform said data recovery instructions and said retransmission.

3. The computer-implemented method of claim 1, wherein said need to regenerate said regenerated records is due to a detection of said at least one missing data record through a failure in a hand-shaking protocol to confirm successful transmission and receipt of at least one data record from said at least one of said relational data stream processing programs to at least one other of the group consisting of:

at least one downstream relational data stream processing programs of said relational data stream processing programs the relational data stream processing program that is downstream of said at least one of said relational data stream processing programs, and an external data consumer or subscriber.

4. The computer-implemented method of claim 1, further comprising eliminating duplicate data records.

5. The computer-implemented method of claim 1, wherein said relational data stream processing program upstream of said location where said need to retransmit has been detected, when such data is no longer available in said buffer of said relational data stream processing program upstream of said location operating independently, regenerates lost data through recursive recovery, wherein said recursive recovery is recursively carried out in said relational data stream processing program upstream of a said location by treating said relational data stream processing program as a downstream relational data stream processing program that also needs input data retransmission from one or more of said relational data stream processing programs that are further upstream of said relational data stream processing program upstream of said a location.

6. A computer-implemented method of recovering a stream processing node when processing streams of data records using a plurality of relational data stream processing programs executing on at least one data processing device and communicating with one another through a plurality of stream processing nodes to form a network of said plurality of relational data stream processing programs, said method comprising:

providing said plurality of relational data stream processing programs executing on said plurality of stream processing nodes for processing said streams of data records;

processing said streams of data records within two or more of said plurality of said relational data stream processing programs according to stream processing operations including relational stream processing operations;

copying a record currently last processed by one of said plurality of stream processing nodes to another one of said plurality of stream processing nodes; and

reinstating processing of said copying record currently last processed in said another one of said plurality of stream processing nodes, using said record having been copied; and

retransmitting previously transmitted records from at least one of said plurality of stream processing nodes upstream of said one of said plurality of stream processing nodes and said another one of said plurality of stream processing nodes, to said another one of said plurality of stream processing nodes.

7. A computer-implemented method of recovering a stream processing node when processing streams of data records using a plurality of relational data stream processing programs executing on at least one data processing device and communicating with one another through a plurality of stream processing nodes to form a network of said plurality of relational data stream processing programs, said method comprising:

providing said plurality of relational data stream processing programs executing on said plurality of stream processing nodes for processing said streams of data records;

processing said streams of data records within two or more of said plurality of said relational data stream processing programs according to stream processing operations including relational stream processing operations, wherein said streams of data records are consumed and generated by said plurality of stream processing nodes as said streams of data records are inputted to and outputted from said plurality of stream processing nodes, respectively; and

upon recovering from a failure of a stream processing node of said plurality of stream processing nodes, receiving retransmission of at least one record from at least one of said plurality of stream processing nodes directly upstream of said stream processing node of said plurality of stream processing nodes having recovered, said at least one record not having been previously acknowledged as being fully processed.

8. The computer-implemented method of claim 7, wherein said at least one of said plurality of stream processing nodes upstream stores copies of records having been outputted from said at least one of said of plurality of stream processing nodes node upstream in a buffer.

9. The computer-implemented method of claim 8, wherein said buffer is configured to be persistent.

10. The computer-implemented method of claim 7, wherein said stream processing node having recovered recovers recursively, wherein said stream processing node having recovered receives retransmission of records from said at least one of said plurality of stream processing nodes upstream;

reprocesses said at least one record received by retransmission; and

regenerates at least one data stream output as a result of the reprocessing.

11. A computer-implemented method of retransmitting data when processing streams of data records using a plurality of relational data stream processing programs executing on at least one data processing device and communicating with one another through data communication interfaces to form a network of said plurality of relational data stream processing programs, said method comprising:

providing said plurality of relational data stream processing programs executing on said at least one data processing device for processing said streams of data records;

processing said streams of data records within a plurality of said relational data stream processing programs according to stream processing operations including relational stream processing operations;

detecting a need to retransmit at least one missing data record of said streams of data records;

sending said at least one missing data record to at least one of said relational data stream processing programs that needs said retransmitting, at a location where said need to retransmit has been detected;

wherein, prior to sending, when said at least one missing data record is available in one or more upstream buffers of at least one relational data stream processing program upstream of the location where said need to retransmit has been detected, said method comprises sending said at least one missing data record from said one or more upstream buffers to the location where said need to retransmit has been detected;

wherein, prior to sending, when all data needed to reconstitute said at least one missing data record are available in said one or more upstream buffers of said at least one relational data stream processing program directly upstream of said location, but at least one of said at least one missing data records is not available in said one or more upstream buffers of said at least one relational data stream processing program directly upstream of said location, said method comprises reconstituting said at least one of said at least one missing data records and sending said at least one missing data record to the location where said need to retransmit has been detected and;

wherein, prior to said sending, when all data needed to reconstitute said at least one missing data record are not available in said one or more upstream buffers of said at least one relational data stream processing program directly upstream of said location, said method comprises regenerating said data through recursive recovery and reconstituting at least one of said at least one missing data records using said data having been regenerated, wherein said recursive recovery is recursively carried out in at least one of said at least one relational data stream processing programs directly upstream, by treating said at least one relational data stream processing program directly upstream as at least one downstream relational data stream processing program that also needs input data retransmission from one or more relational data stream processing programs that are further upstream of said at least one relational data stream processing program directly upstream.

12. A computer-implemented method of retransmitting data when processing streams of data records using a plurality of relational data stream processing programs executing on at least one data processing device and communicating with one another through data communication interfaces to form a network of said plurality of relational data stream processing programs, said method comprising:

providing said plurality of relational data stream processing programs executing on said at least one data processing device for processing said streams of data records;

processing said streams of data records within two or more of said plurality of relational data stream processing programs according to stream processing operations including relational stream processing operations;

detecting a need to retransmit at least one missing data record of said streams of data records; and

sending said at least one missing data record to a location where said need to transmit has been detected;

wherein, prior to said sending, when all data needed to reconstitute said at least one missing data record are available in said one or more buffers of at least one relational data stream processing program directly upstream of said location, but at least one of said at least one missing records is not available, said method comprises reconstituting said at least one of said at least one missing data records and sending said at least one missing data record to the location where said need to transmit has been detected.

13. A computer-implemented method of retransmitting data when processing streams of data records using a plurality of relational data stream processing programs executing on at least one data processing device and communicating with one another through data communication interfaces to form a network of said plurality of relational data stream processing programs, said method comprising:

providing said plurality of relational data stream processing programs executing on said at least one data processing device for processing said streams of data records;

processing said streams of data records within two or more of said plurality of relational data stream processing programs according to stream processing operations including relational stream processing operations;

detecting a need to retransmit at least one missing data record of said streams of data records; and

sending said at least one missing data record to a location where said need to transmit has been detected;

wherein, prior to said sending, when all data needed to reconstitute said at least one missing data record are not available in one or more buffers of at least one relational data stream processing program directly upstream of said location, said method comprises regenerating said data through recursive recovery including reconstituting said at least one missing data record using said regenerated data, wherein said recursive recovery is recursively carried out in said at least one relational data stream processing program directly upstream, by treating said at least one relational data stream processing program directly upstream as at least one downstream relational data stream processing program that also needs input data retransmission from one or more of said plurality of relational data stream processing programs that are further upstream of said at least one relational data stream processing program directly upstream.

14. A computer-implemented method of retransmitting data when processing streams of data records using a plurality of relational data stream processing programs executing on at least one data processing device and communicating with one another through data communication interfaces to form a network of said plurality of relational data stream processing programs, said method comprising:

providing said plurality of relational data stream processing programs executing on said at least one data processing device for processing said streams of data records;

processing said streams of data records within two or more of said plurality of relational data stream processing programs according to stream processing operations including relational stream processing operations;

detecting a need to retransmit at least one missing data record of said streams of data records; and

sending said at least one missing data record to a location where said need to transmit has been detected;

wherein, prior to said sending, when all data needed to reconstitute said at least one missing data record are available in said one or more buffers of at least one relational data stream processing program directly upstream of said location, but at least one of said at least one missing records is not available in said one or more buffers of said at least one relational data stream processing program directly upstream of said location, said method comprises reconstituting said at least one of said at least one missing data records and sending said at least one missing data record to the location where said need to transmit has been detected; and

wherein, prior to said sending, when all data needed to reconstitute said at least one missing data record are not available in one or more buffers of at least one relational data stream processing program directly upstream of said location, said method comprises regenerating said data through recursive recovery including reconstituting said at least one missing data record using said regenerated data, wherein said recursive recovery is recursively carried out in at least one relational data stream processing program directly upstream, by treating said at least one relational data stream processing program directly upstream as at least one downstream relational data stream processing program that also needs input data retransmission from one or more relational data stream processing programs that are further upstream of said at least one relational data stream processing program directly upstream.