Continuous processing language for real-time data streams

• US 8,396,886 B1
• Filed: 02/02/2006
• Issued: 03/12/2013
• Est. Priority Date: 02/03/2005
• Status: Active Grant

• Current Assignee: Sybase Incorporated (SAP SE)
• Original Assignee: Sybase Incorporated (SAP SE)
• Inventors: Tsimelzon, Mark, Sanin, Aleksey, Motwani, Rajeev, Seidman, Glenn Robert, Patel, Gayatri
• Primary Examiner(s): Shmatov, Alexey
• Application Number:
  US11/346,119
• Time in Patent Office: 2,595 Days
• Field of Search: 707/100, 707/999.1, 707/769
• US Class Current: 707/769
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/2458 Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries

Abstract

A computer software language capable of expressing registered queries that operate on one more or more data streams continuously. The language of the present invention is based on a publish/subscribe model in that queries subscribe to data streams and publish to data streams. Also, the language of the present invention can express queries that operate directly on data streams. Since queries expressed in the language of the present invention may be executed continuously and directly on data streams, the language includes a clause for specifying time-based and/or row-based windows for the input data stream. Operations are then performed on the data within such windows. In one embodiment, the language is also SQL-like and includes a clause for defining named windows (which can be used in any number of queries); a clause for detecting a pattern, and correlated database subqueries for correlating data stream data with database tables.

Claims

1. A computer-implemented method in a continuous processing system, comprising:

subscribing to one or more continuous input data streams from one or more publishers of data streams, wherein each of the one or more continuous input data streams is published in a publisher-specific data representation format via a publisher-specific remote communication protocol;

executing a query operation directly on the one or more continuous input data streams, including any windows of data within the one or more continuous input data streams, without any pre-processing of the continuous input data streams, including without storing the continuous input data streams in any data structure, prior to said executing, wherein the query operation is executed on a continuous basis; and

publishing, in real time and on a continuous basis, a result of the query operation to one or more output data streams in a subscriber-specific data representation format via a subscriber-specific remote communication protocol as subscribed to by one or more subscribers, wherein the publisher-specific remote communication protocol varies from the subscriber-specific remote communication protocol.

2. The computer-implemented method of claim 1, wherein the executing further comprises:

executing a first query operation directly on a first continuous input data stream without any pre-processing of the first continuous input data stream prior to said executing, wherein the first query operation is executed on a continuous basis;

subscribing to a second continuous input data stream from the one or more publishers of data streams, wherein the second continuous input data stream is published in a second publisher-specific data representation format via a second publisher-specific remote communication protocol;

executing a second query operation directly on the second continuous input data stream without any pre-processing of the second continuous input data stream prior to said executing, wherein the second query operation is executed on a continuous basis; and

combining a first result of the first query operation with a second result of the second query operation to produce the result.

3. The computer-implemented method of claim 1, wherein the executing a query operation comprises:

executing the query operation directly on the one or more continuous input data streams without loading said continuous data input stream into a table prior to said executing.

4. The computer-implemented method of claim 1, wherein the executing a query operation comprises:

executing the query operation directly on the one or more continuous input data streams in said publisher-specific data representation format prior to said executing.

5. A non-transitory computer-readable medium having computer-executable instructions stored thereon that, when executed by a computing device, cause the computing device to perform a method comprising:

subscribing to one or more continuous input data streams from one or more publishers of data streams, wherein each of the one or more continuous input data streams is published in a publisher-specific data representation format via a publisher-specific remote communication protocol;

executing a query operation directly on the one or more continuous input data streams, including any windows of data within the one or more continuous input data streams, without any pre-processing of the continuous input data streams, including without storing the continuous input data streams in any data structure, prior to said executing, wherein the query operation is executed on a continuous basis; and

publishing, in real time and on a continuous basis, a result of the query operation to one or more output data streams in a subscriber-specific data representation format via a subscriber-specific remote communication protocol as subscribed to by one or more subscribers, wherein the publisher-specific remote communication protocol varies from the subscriber-specific remote communication protocol.

6. The non-transitory computer-readable medium of claim 5, wherein the executing further comprises:

executing a first query operation directly on a first continuous input data stream without any pre-processing of the first continuous input data stream prior to said executing, wherein the first query operation is executed on a continuous basis;

subscribing to a second continuous input data stream from the one or more publishers of data streams;

executing a second query operation directly on a second continuous input data stream without any pre-processing of the second continuous input data stream prior to said executing, wherein the second query operation is executed on a continuous basis;

combining a first result of the first query operation with a second result of the second query operation to produce an output; and

wherein publishing further comprises publishing, in real time and on a continuous basis, the output to the one or more output data streams.

7. A system comprising at least one processor and one memory, the processor including one or more units operating thereon and in conjunction with said memory, the one or more units comprising:

a publication manager to subscribe to one or more continuous input data streams from one or more publishers of data streams, wherein each of the one or more continuous input data streams is published in a publisher-specific data representation format via a publisher-specific remote communication protocol;

a query module manager to execute a query operation directly on the one or more continuous input data streams, including any windows of data within the one or more continuous input data streams, without any pre-processing of the continuous input data streams, including without storing the continuous input data streams in any data structure, prior to said executing, wherein the query operation is executed on a continuous basis; and

a subscription manager to publish, in real time and on a continuous basis, a result of the query operation to one or more output data streams in a subscriber-specific data representation format via a subscriber-specific remote communication protocol as subscribed to by one or more subscribers, wherein the publisher-specific remote communication protocol differs from the subscriber-specific remote communication protocol.

8. The system of claim 7, further comprising:

an adaptor manager to interpret the publisher-specific data representation format and to communicate with each of the one or more publishers of data streams using a respective publisher-specific remote communication protocol.

9. The system of claim 8, wherein the publication manager is configured to subscribe to a first continuous input data stream from a first publisher of data streams, and wherein the first continuous input data stream is published in a data representation format via a remote communication protocol that are specific to said first publisher.

10. The system of claim 9, wherein the adaptor manager comprises a first publisher adaptor to interpret the data representation format specific to said first publisher and to communicate with said first publisher using the remote communication protocol specific to said first publisher.

11. The system of claim 10, wherein the publication manager is further configured to subscribe to a second continuous input data stream from a second publisher of data streams, and wherein the second continuous input data stream is published in a data representation format via a remote communication protocol that are specific to said second publisher.

12. The system of claim 11, wherein the adaptor manager further comprises a second publisher adaptor to interpret the data representation format specific to said second publisher and to communicate with said second publisher using the remote communication protocol specific to said second publisher, wherein the data representation format specific to said second publisher differs from the data representation format specific to said first publisher, and wherein the remote communication protocol specific to said second publisher differs from the remote communication protocol specific to said first publisher.

13. The system of claim 12, wherein the subscription manager is configured to publish, in real time and on a continuous basis, a result of the query operation to a first output data stream, wherein a first subscriber subscribes to the first output data stream, and wherein the first output data stream is published in a data representation format via a remote communication protocol that are specific to said first subscriber.

14. The system of claim 13, wherein the adaptor manager further comprises a first subscriber adaptor to transform the result to the data representation format specific to said first subscriber and to communicate with said first subscriber using the first subscriber-specific remote communication protocol.

15. The system of claim 14, wherein the subscription manager is further configured to publish, in real time and on a continuous basis, a result of the query operation to a second output data stream, wherein a second subscriber subscribes to the second output data stream, and wherein the second output data stream is published in a data representation format via a remote communication protocol that are specific to said second subscriber.

16. The system of claim 15, wherein the adaptor manager further comprises a second subscriber adaptor to transform the result to the data representation format specific to said second subscriber and to communicate with said second subscriber using the remote communication protocol specific to said second subscriber, wherein the data representation format specific to said second subscriber differs from the data representation format specific to said first subscriber, and wherein the remote communication protocol specific to said second subscriber differs from the remote communication protocol specific to said first subscriber.

17. The system of claim 7, wherein the query module manager is further configured to

execute a first query operation directly on a first continuous input data stream,

further configured to execute a second query operation directly on a second continuous input data stream, and further configured to combine a first result of the first query operation with a second result of the second query operation to produce an output, wherein the first and second query operations are executed on a continuous basis without any pre-processing of the first continuous input data stream or the second continuous input data stream prior to said executing the first query and the second query, and

wherein the subscription manager is configured to publish, in real time and on a continuous basis, the output to one or more output data streams.

18. The system of claim 17, further comprising:

an adaptor manager to interpret the first and second publisher-specific data representation formats and to communicate with the first and second publishers of data streams using the respective first and second publisher-specific remote communication protocols.

19. The system of claim 18, wherein the subscription manager is configured to publish, in real time and on a continuous basis, the output to a first output data stream, wherein a first subscriber subscribes to the first output data stream, and wherein the first output data stream is published in a data representation format via a remote communication protocol that are specific to said first subscriber.

20. The system of claim 19, wherein the adaptor manager further comprises a first subscriber adaptor to transform the output to the data representation format specific to said first subscriber and to communicate with said first subscriber using the remote communication protocol specific to said first subscriber.

21. The system of claim 20, wherein the subscription manager is further configured to publish, in real time and on a continuous basis, the output to a second output data stream, wherein a second subscriber subscribes to the second output data stream, and wherein the second output data stream is published in a data representation format via a remote communication protocol that are specific to said second subscriber.

22. The system of claim 21, wherein the adaptor manager further comprises a second subscriber adaptor to transform the output to the data representation format specific to said second subscriber and to communicate with said second subscriber using the remote communication protocol specific to said second subscriber, wherein the data representation format specific to said second subscriber differs from the data representation format specific to said first subscriber, and wherein the remote communication protocol specific to said second subscriber differs from the remote communication protocol specific to said first subscriber.