COMPLEX EVENT PROCESSOR FOR HISTORIC/LIVE/REPLAYED DATA

• US 20160292016A1
• Filed: 04/02/2015
• Published: 10/06/2016
• Est. Priority Date: 04/02/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bussard, Laurent, Santos, Ivo Jos Garcia dos, Goldstein, Jonathan, Nano, Olivier, Tarnavski, Tihomir, Chandramouli, Badrish, Novik, Lev
• Granted Patent: US 9,632,846 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/2474 Sequence data queries, e.g. querying versioned data
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 9/54 Interprogram communication
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications

Abstract

A complex event processor is described which has a communications interface configured to retrieve event data by pulling it from one or more sources and to receive at least one live event stream pushed to the interface. An event processing pipeline connected to the communications interface comprises a plurality of operator nodes connected between the communications interface and a combiner node which is a node configured to combine event data from the source and from the live event stream. The communications interface is configured to retrieve events from the source and to push the events retrieved from the source along the event processing pipeline towards the combiner node. The communications interface is configured to retrieve and push the retrieved events only in response to request messages passed in an upstream direction from the combiner node to the communications interface along the pipeline.

Claims

1. A system comprising:

at least one processor adapted to execute computer executable instructions associated with nodes of an event processing pipeline;

a communications interface communicatively coupled to the at least one processor and configured to;

pull first event data from at least one source in response to a request message; and

receive second event data pushed to the communications interface, the second event data associated with at least one live/replayed event stream; and

at least one computer-readable media communicatively coupled to the at least one processor and having thereon a plurality of nodes, the plurality of nodes comprising;

a first group of operator node(s) connected between the communications interface and the combiner node, the first group of operator node(s) configured to provide third event data based at least in part on the first event data;

a second group of operator node(s) connected between the communications interface and the combiner node, the second group of operator node(s) configured to provide fourth event data based at least in part on the second event data; and

a combiner node configured to;

determine a request time based at least in part on at least some of the fourth event data;

provide the request message comprising the request time to the communications interface via the first group of operator node(s); and

combine at least some of the third event data and at least some of the fourth event data.

2. The system of claim 1, wherein the combiner node is configured to combine the at least some of the third event data and the at least some of the fourth event data by performing at least one of a join, temporal join, or union of the at least some of the third event data and the at least some of the fourth event data.

3. The system of claim 1, wherein the combiner node is further configured to determine a timestamp associated with at least some of the fourth event data and to determine the request time based at least in part on the timestamp.

4. The system of claim 1, wherein an operator node of the operator node(s) of the first group of operator node(s) is configured to:

determine an altered request time based at least in part on the request time in the request message;

determine an altered request message based at least in part on the request message and the altered request time; and

provide the altered request message to the communications interface in place of the request message.

5. The system of claim 4, wherein at least one of the one or more of the operator node(s) is configured to alter the request time based at least in part on a stored inverse function.

7. The system of claim 1, wherein:

the source is configured to store the first event data associated with an interval index;

the request message comprises one or more times to be used to retrieve point and/or interval events from the store; and

an interval event comprises a start edge event and an end edge event.

8. The system of claim 1, wherein the request message further comprises at least one of data of a number of events of the first event data to be retrieved from the source or data of a volume of the first event data to be retrieved from the source.

9. The system of claim 8, wherein one or more of the operator node(s) of at least one of the first group of operator node(s) or the second group of operator node(s) are configured to modify the number of events to be retrieved from the source based at least in part on operations defined at the operator nodes.

10. The system of claim 1, wherein at least one operator node of the first group of operator node(s) is configured to:

receive the request message;

determine event data to be retrieved based at least in part on a difference between a record of retrieved event data received at the at least one operator node and event data requested by the request message; and

generate and send an outgoing request message towards the communications interface, the outgoing request message requesting at least some of the determined event data to be retrieved.

11. The system of claim 1, wherein at least one operator node of at least one of the first group of operator node(s) or the second group of operator node(s) is configured to store a record of events received at the at least one operator node.

12. The system of claim 1, wherein at least one operator node of the first group of operator node(s) is configured to:

store a record of request messages received at the at least one operator node; and

receive the request message; and

transmit the request message towards the communications interface based at least in part on the record indicating the request message is not a duplicate request message.

13. The system of claim 1, wherein the communications interface is further configured to form batches of the first event data.

6. (canceled)

14. A method comprising:

receiving at least one live/replayed event stream;

pushing data of live/replayed events of the live/replayed event stream to a combiner node in an event processing pipeline;

determining, at the combiner node, a request time based at least in part on at least some of the data of the live/replayed events;

providing a request message comprising the request time from the combiner node to a communications interface;

retrieving data of historical events from at least one event source using a communications interface in response to the request message; and

combining, at the combiner node, at least some of the data of the live/replayed events and at least some of the data of the historical events.

15. The method of claim 14, further comprising providing the request message from the combiner node to the communications interface via an operator node, and at the operator node:

determining an altered request time based at least in part on the request time in the request message;

generating and sending an outgoing request message to the communications interface in place of the request message, the outgoing request message comprising the altered request time.

16. The method of claim 14, further comprising retrieving point and/or interval events from the event source based at least in part on the request time.

17. The method of claim 14, further comprising:

providing the request message from the combiner node to the communications interface via an operator node; and

at the operator node, on receipt of the request message;

determining event data to be retrieved based at least in part on a difference between a record of retrieved event data received at the operator node and retrieved event data requested by the incoming request message, and

generating and sending an outgoing request message towards the communications interface, the outgoing request message requesting at least some of the determined event data to be retrieved.

18. The method of claim 14, the request message being one of a plurality of request messages and the method further comprising providing the plurality of request messages from the combiner node to the communications interface via an operator node, and at the operator node:

storing a record of request messages of the plurality of request messages received at the operator node;

receiving the request message; and

transmitting the request message towards the communications interface based at least in part on the record indicating the request message is not a duplicate request message.

19. One or more computer storage media storing computer-executable instructions that, when executed by a computing system, direct the computing system to perform steps comprising:

pushing live/replayed events from a communications interface to a combiner node of an event processing pipeline;

retrieving events from at least one event source based at least in part on the live/replayed events and results of a static analysis of one or more operator nodes connected between the communications interface and the combiner node;

pushing the retrieved events to the one or more operator nodes, wherein the one or more operator nodes provide first events based at least in part on the retrieved events; and

combining the live/replayed event stream and the first events at the combiner node.

20. The one or more computer storage media of claim 20, the steps further comprising:

retrieving the events based further upon predetermined amounts and/or timestamp criteria of retrieved event data.

21. The one or more computer storage media of claim 20, the steps further comprising:

sending, by the combiner node, a request message including a request time based at least in part on at least some of the live/replayed events; and

determining, by an operator node of the one or more operator nodes, an adjusted time based at least in part on the request time;

determining, by the operator node, an adjusted request message including the adjusted request time;

retrieving second events from the at least one event source based at least in part on the adjusted request time; and

combining, by the combiner node, the live/replayed event stream and the retrieved second events.