Method for distributed RDSMS

• US 9,049,196 B1
• Filed: 06/30/2014
• Issued: 06/02/2015
• Est. Priority Date: 03/15/2003
• Status: Active Grant

• Current Assignee: Guavus, Inc. (Thales SA)
• Original Assignee: SQLstream, Inc. (Thales SA)
• Inventors: Black, Damian
• Primary Examiner(s): Ly, Anh
• Application Number:
  US14/318,772
• Time in Patent Office: 337 Days
• Field of Search: 707/741, 707/722, 707/713, 707/769, 707/770, 707/802, 707/692, 707/773, 707/805, 707/674, 707/644, 707/E17.055, 707/E17.002, 709/237, 709/230, 709/231, 715/757, 715/810, 715/753, 455/405
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/83 Querying
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: Y10S 707/922 Communications
• CPC Class Codes: Y10S 707/954 Relational
• CPC Class Codes: Y10S 707/959 Network
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

Methods for processing streaming data records are provided that include facilities for defining and naming multiple input and output data streams using relational processing definitions and abstractions similar to those found in relational database management systems (RDBMS) and embodied as a set of communicating stream processing nodes. The result is a Distributed Data Stream Management System (DDSMS) which is also a kind of Relational Data Stream Management System (RDSMS). The former comprises a set of Relational Data Stream Management System (RDSMS) nodes, whereas the latter might contain only a single RDSMS node.

Claims

1. A distributed data stream management system comprising:

a network of communicating relational stream processing computer nodes distributed and configured for processing streams of data records received from at least one external source external of said system, according to relational stream processing operations, without a need to include a persistent relational database element to process said streams of data records, and outputting processed streams of data records to at least one external consumer external of said system;

at least two of said relational stream processing computer nodes each comprising;

an input interface configured to receive input streams of said streams of data records; and

an output interface configured to transmit output streams of said streams of data records;

a central configuration server operably connectable to said relational stream processing computer nodes to communicate with said relational stream processing computer nodes;

wherein said streams of data records received from at least one external source are broken up into component pieces and said component pieces are allocated to different ones of said relational stream processing computer nodes for said processing;

wherein said at least two of said relational stream processing computer nodes aggregate a plurality of said output streams into a consumer output stream of said streams of records and output said consumer output stream to at least one of said at least one external consumer;

wherein said component pieces are managed by said relational stream processing nodes, respectively, in retransmission buffers, and said relational stream processing nodes dispose of said managed component pieces upon receipt of acknowledgement from a predefined subset of said at least one external consumer that said consumer output stream has been successfully received; and

wherein said stream processing computer nodes operate in a functionally equivalent manner as if a single stream processing computer node processed said streams of data records received from at least one external source.

2. The distributed data stream management system of claim 1, wherein said relational stream processing computer nodes each include a configuration interface, and wherein said central configuration server is connectable to said relational stream processing nodes through said configuration interfaces to deliver configuration information thereto.

3. The distributed data stream management system of claim 2, wherein configurations of said relational stream processing computer nodes are dynamically updatable by delivery of said configuration information from said central configuration server.

4. The distributed data stream management system of claim 1, wherein said relational stream processing computer nodes each comprise a control interface configured to receive signals for controlling a speed of transmission of said output streams of said streams of data records.

5. The distributed data stream management system of claim 1, wherein said relational stream processing computer nodes and said central configuration server comprise logical servers that are mappable onto one or more computer servers or systems.

6. The distributed data stream management system of claim 1, wherein said at least one external source comprises sensors selected from at least one of the members of the group consisting of:

location based devices, environment sensors, machinery status sensors, plant equipment sensors, engine sensors, vehicle sensors and instrument sensors.

7. The distributed data stream management system of claim 1, wherein said at least one external source comprises distributed service components and said streams of data records are processed by said system to aggregate down, filter and construct transaction records for analysis or billing.

8. The distributed data stream management system of claim 1, wherein said consumer output stream is configured for monitoring a network selected from at least one of the network groups consisting of:

electronic data, mobile, wireless, electricity, water, highway, airway, railway and waterway.

9. The distributed data stream management system of claim 1, wherein said consumer output stream is configured for facilitating at least one of:

fraud detection, surveillance, prepaid service usage billing, postpaid service usage billing and billing analysis.

10. A method of monitoring comprising:

providing a network of communicating relational stream processing computer nodes distributed and configured for processing streams of data records;

receiving said streams of data records from at least one sensor;

breaking up said streams of data records into component substreams and allocating said component substreams to different ones of said relational stream processing computer nodes;

processing said component substreams by said different ones of said relational stream processing computer nodes according to relational stream processing operations;

aggregating a plurality of output streams, outputted by said different ones of said relational stream processing computer nodes, resulting from said processing; and

outputting an aggregated stream resulting from said aggregating to at least one consumer;

wherein said component substreams are managed by said relational stream processing nodes, respectively, in retransmission buffers, and said relational stream processing nodes dispose of said managed component substreams upon receipt of acknowledgement from a predefined subset of said at least one consumer that said aggregated stream has been successfully received.

11. The method of claim 10, wherein said at least one sensor is selected from at least one of the members of the group consisting of:

location based devices, environment sensors, machinery status sensors, plant equipment sensors, engine sensors, vehicle sensors and instrument sensors.

12. A method of processing streams of data records, said method comprising:

providing a network of communicating relational stream processing computer nodes distributed and configured for processing streams of data records;

receiving said streams of data records from at least one source;

breaking up a set of relational stream processing operations to be performed on said streams of data records into component pieces of said set and allocating said component pieces to different ones of said relational stream processing computer nodes;

processing said streams of data by said different ones of said relational stream processing nodes according to said component pieces of said set of relational stream processing operations assigned to each of said different ones of said relational stream processing nodes, respectively;

aggregating a plurality of output streams, outputted by said different ones of said relational stream processing computer nodes, resulting from said processing; and

outputting an aggregated stream resulting from said aggregating to at least one a consumer;

wherein said component pieces are managed by said relational stream processing nodes, respectively, in retransmission buffers, and said relational stream processing nodes dispose of said managed component pieces upon receipt of acknowledgement from a predefined subset of said at least one consumer that said aggregated stream has been successfully received.

13. The method of claim 12, performed without a need to include a persistent relational database element to process said component pieces.

14. The method of claim 12, wherein said breaking up said set of relational stream processing operations and said allocating are performed in a manner to optimize processing performance.

15. The method of claim 12, further comprising dynamically optimizing performance of the method, wherein said dynamically optimizing comprises at least one of restructuring said component pieces and reallocating said component pieces or restructured component pieces during said processing.

16. The method of claim 12, wherein stream definitions for the relational stream processing computer nodes are held in a central configuration store server;

wherein the central configuration store server communicates with the relational stream processing computer nodes; and

wherein the relational stream processing computer nodes perform said processing according to the stream definitions applied thereto.

17. The method of claim 16, further comprising the central configuration store server broadcasting changes to stream definitions to all relational stream processing computer nodes in the network.

18. The method of claim 16, further comprising the central configuration store server transmitting changes to stream definitions to a select set of relational stream processing computer nodes in the network, wherein said select set is less than all of the relational stream processing computer nodes in the network.

19. The method of claim 12, wherein said aggregating a plurality of output streams comprises locally aggregating by performing multiple aggregations over multiple ones of said relational stream processing nodes each receiving multiple output streams, to reduce stream volume, and aggregating results of the local aggregations by another one of said relational stream processing nodes to provide one or more final, aggregated output streams.

20. The method of claim 12, further comprising analyzing said processing said streams of data by said different ones of said relational stream processing nodes and, for ones of said relational stream processing nodes that yield no results, controlling said ones of said relational stream processing nodes that yield no results to discontinue processing the streams of data or not generate an output stream.