Event batching, output sequencing, and log based state storage in continuous query processing
First Claim
1. A system comprising:
- a memory storing a plurality of instructions; and
one or more processors configured to access the memory, wherein the one or more processors are further configured to execute the plurality of instructions to at least;
receive a continuous input stream of events related to an application;
generate a checkpoint marker event;
determine an interval for inserting the checkpoint marker event, the interval based at least in part on a type of the application, a latency requirement of the application, and a frequency at which events of the input stream of events are received;
insert the checkpoint marker event into the continuous input stream of events at each interval;
generate a set of one or more event batches based at least in part on the checkpoint marker event;
determine a size for a particular event batch of the one or more event batches based at least in part on the interval for inserting the checkpoint marker event;
process the continuous input stream of events to generate an output stream of events related to the application;
determine an output sequence number for an output event in the output stream of events;
transmit the output event comprising the output stream of events;
store the output sequence number of the output event;
while the continuous input stream of events is being processed, receive an indication of failure of the system; and
in response to the indication of failure;
determine a current output sequence number of a most recently transmitted output event in the output stream of events;
determine a last output sequence number of an output event corresponding to a most recently processed batch of the one or more event batches;
determine a set of one or more output events of the output stream to be transmitted based at least in part on the current sequence number and the last output sequence number; and
transmit the set of one or more output events related to the application.
1 Assignment
0 Petitions
Accused Products
Abstract
An event processing system is disclosed that processes events of an event stream and performs the recovery of events during system failure. The recovery of events is performed by generating output sequence numbers for each deterministic output event generated by the event processing system. In an embodiment, the event processing system determines a current output sequence number of a most recently transmitted output event in a output stream of events, a last output sequence number of an output event corresponding to a most recently processed batch of input events and determines a set of one or more output events of the output stream to be transmitted based on the current sequence number and the last output sequence number. The event processing system then transmits the output events related to the application.
27 Citations
18 Claims
-
1. A system comprising:
-
a memory storing a plurality of instructions; and one or more processors configured to access the memory, wherein the one or more processors are further configured to execute the plurality of instructions to at least; receive a continuous input stream of events related to an application; generate a checkpoint marker event; determine an interval for inserting the checkpoint marker event, the interval based at least in part on a type of the application, a latency requirement of the application, and a frequency at which events of the input stream of events are received; insert the checkpoint marker event into the continuous input stream of events at each interval; generate a set of one or more event batches based at least in part on the checkpoint marker event; determine a size for a particular event batch of the one or more event batches based at least in part on the interval for inserting the checkpoint marker event; process the continuous input stream of events to generate an output stream of events related to the application; determine an output sequence number for an output event in the output stream of events; transmit the output event comprising the output stream of events; store the output sequence number of the output event; while the continuous input stream of events is being processed, receive an indication of failure of the system; and in response to the indication of failure; determine a current output sequence number of a most recently transmitted output event in the output stream of events; determine a last output sequence number of an output event corresponding to a most recently processed batch of the one or more event batches; determine a set of one or more output events of the output stream to be transmitted based at least in part on the current sequence number and the last output sequence number; and transmit the set of one or more output events related to the application. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8)
-
-
9. A method comprising:
-
receiving a continuous input stream of events related to an application; generating a checkpoint marker event; determining an interval for inserting the checkpoint marker event, the interval based at least in part on a type of the application, a latency requirement of the application, and a frequency at which events of the input stream of events are received; inserting the checkpoint marker event into the continuous input stream of events at each interval; generating a set of one or more event batches based at least in part on the checkpoint marker event; determining a size for a particular event batch of the one or more event batches based at least in part on the interval for inserting the checkpoint marker event; processing the continuous input stream of events to generate an output stream of events related to the application; determining an output sequence number for an output event in the output stream of events; transmitting the output event of the output stream of events; storing the output sequence number of the output event; while the continuous input stream of events is being processed, receiving an indication of failure of the system; and in response to the indication of failure; determining a current output sequence number of a most recently transmitted output event in the output stream of events; determining a last output sequence number of an output event corresponding to a most recently processed batch of the one or more event batches; determining a set of one or more output events of the output stream to be transmitted based at least in part on the current sequence number and the last output sequence number; and transmitting the set of one or more output events related to the application. - View Dependent Claims (10, 11, 12, 13, 14)
-
-
15. A non-transitory computer-readable storage memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising:
-
instructions that cause the one or more processors to receive a continuous input stream of events related to an application; instructions that cause the one or more processors to generate a checkpoint marker event; instructions that cause the one or more processors to determine an interval for inserting the checkpoint marker event, the interval based at least in part on a type of the application and a latency requirement of the application; instructions that cause the one or more processors to insert the checkpoint marker event into the continuous input stream of events at each interval; instructions that cause the one or more processors to generate a set of one or more event batches based at least in part on the checkpoint marker event; instructions that cause the one or more processors to determine a size for a particular event batch of the one or more event batches based at least in part on the interval for inserting the checkpoint marker event; instructions that cause the one or more processors to process the continuous input stream of events to generate an output stream of events related to the application; instructions that cause the one or more processors to determine an output sequence number for an output event in the output stream of events; instructions that cause the one or more processors to transmit the output event of the output stream of events; instructions that cause the one or more processors to store the output sequence number of the output event; instructions that cause the one or more processors to receive an indication of failure of the system while the continuous input stream of events is being processed; instructions that cause the one or more processors to determine a current output sequence number of a most recently transmitted output event in the output stream of events; instructions that cause the one or more processors to determine a last output sequence number of an output event corresponding to a most recently processed batch of the one or more event batches; instructions that cause the one or more processors to determine a set of one or more output events of the output stream to be transmitted based at least in part on the current sequence number and the last output sequence number; and instructions that cause the one or more processors to transmit the set of one or more output events related to the application. - View Dependent Claims (16, 17, 18)
-
Specification