Event batching, output sequencing, and log based state storage in continuous query processing

• US 10,127,120 B2
• Filed: 05/20/2016
• Issued: 11/13/2018
• Est. Priority Date: 10/22/2015
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Park, Hoyong, Bishnoi, Sandeep, Deshmukh, Unmesh Anil, Thukkaram, Prabhu
• Primary Examiner(s): Hoang, Son T
• Application Number:
  US15/161,062
• Publication Number:
  US 20170116210A1
• Time in Patent Office: 907 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/2433 Query languages
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications

Abstract

An event processing system is disclosed that processes events of an event stream, performs the recovery of events during system failure and preserves the state of the system reliably and accurately while achieving desired system performance. In an embodiment, the event processing system processes a first batch of events of a continuous input stream of events using a continuous query and generates an output stream of events related to an application. The event processing system identifies one or more operators of the continuous query and determines that an operator is a journaled operator. The event processing system generates a journaled snapshot of a current state of the system based on execution of the journaled operator on at least the first batch of events and stores the journaled snapshot of the current state of the system.

Claims

1. A system comprising:

a memory storing a plurality of instructions; and

one or more hardware processors configured to access the memory, wherein the one or more hardware processors are further configured to execute the plurality of instructions to at least;

receive a continuous input stream of events related to an application;

process a first batch of events of the continuous input stream of events using a continuous query to generate an output stream of events related to the application;

identify one or more operators of the continuous query;

determine that an operator type of the identified one or more operators is a journaled operator;

generate a journaled snapshot of a current state of the system based at least in part on execution of the journaled operator on at least the first batch of events; and

store the journaled snapshot of the current state of the system.

2. The system of claim 1, wherein the one or more processors are further configured to execute the plurality of instructions to at least:

determine that the operator type of the identified one or more operators is a non-journaled operator;

generate a full snapshot of the current state of the system based at least in part on execution of the non-journaled operator on at least the first batch of events; and

store the full snapshot of the current state of the system.

3. The system of claim 1, wherein the one or more processors are further configured to execute the plurality of instructions to at least:

generate a checkpoint marker event;

insert the checkpoint marker event into the continuous input stream of events; and

generate at least the first batch of events based at least in part on the checkpoint marker event.

4. The system of claim 1, wherein the one or more processors are further configured to execute the plurality of instructions to at least:

trigger generation of a full snapshot of the current state of the system after a pre-determined number of event batches have been processed.

5. The system of claim 1, wherein the one or more processors are further configured to execute the plurality of instructions to at least:

determine at least one of synopses information or mutable state information related to execution of the one or more operators identified in the continuous query.

6. The system of claim 5, wherein the snapshot of the current state of the system comprises the synopses state information or the mutable state information related to execution of the one or more operators identified in the continuous query.

7. The system of claim 1, wherein the one or more processors are further configured to execute the plurality of instructions to at least determine synopses state information related to execution of the journaled operator identified in the continuous query.

8. The system of claim 1, wherein the one or more processors are further configured to execute the plurality of instructions to at least determine mutable state information related to execution of a non-journaled operator identified in the continuous query.

9. The system of claim 1, wherein the snapshot of the current state of the system comprises information related to at least one of an input state, an operator state, or an output state related to execution of the one or more operators identified in the continuous query.

10. A method, implemented at least by one computing device, comprising:

receiving a continuous input stream of events related to an application;

processing a first batch of events of the continuous input stream of events using a continuous query to generate an output stream of events related to the application;

identifying one or more operators of the continuous query;

determining that an operator type of the identified one or more operators is a journaled operator;

generating a journaled snapshot of a current state of the system based at least in part on execution of the journaled operator on at least the first batch of events; and

storing the journaled snapshot of the current state of the system.

11. The method of claim 10, further comprising:

determining that the operator type of the identified one or more operators is a non-journaled operator;

generating a full snapshot of the current state of the system based at least in part on execution of the non-journaled operator on at least the first batch of events; and

storing the full snapshot of the current state of the system.

12. The method of claim 10, further comprising:

generating a checkpoint marker event;

inserting the checkpoint marker event into the continuous input stream of events; and

generating at least the first batch of events based at least in part on the checkpoint marker event.

13. The method of claim 10, further comprising triggering generation of a full snapshot of the current state of the system after a pre-determined number of event batches have been processed.

14. The method of claim 10, further comprising determining at least one of synopses information or mutable state information related to execution of the one or more operators identified in the continuous query.

15. The method of claim 14, wherein the snapshot of the current state of the system comprises the synopsis state information or the mutable state information related to execution of the one or more operators identified in the continuous query.

16. The method of claim 10, further comprising determining synopses state information related to execution of the journaled operator identified in the continuous query.

17. A non-transitory computer-readable storage memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising:

instructions that cause the one or more processors to receive a continuous input stream of events related to an application;

instructions that cause the one or more processors to process a first batch of events of the continuous input stream of events using a continuous query to generate an output stream of events related to the application;

instructions that cause the one or more processors to identify one or more operators of the continuous query;

instructions that cause the one or more processors to determine that an operator type of the identified one or more operators is a journaled operator;

instructions that cause the one or more processors to generate a journaled snapshot of a current state of the system based at least in part on execution of the journaled operator on at least the first batch of events; and

instructions that cause the one or more processors to store the journaled snapshot of the current state of the system.

18. The non-transitory computer-readable storage memory of claim 17, wherein the plurality of instructions further comprise:

instructions that cause the one or more processors to determine that the operator type of the identified one or more operators is a non-journaled operator;

instructions that cause the one or more processors to generate a full snapshot of the current state of the system based at least in part on execution of the non-journaled operator on at least the first batch of events; and

instructions that cause the one or more processors to store the full snapshot of the current state of the system.

19. The non-transitory computer-readable storage memory of claim 17, wherein the plurality of instructions further comprise:

instructions that cause the one or more processors to generate a checkpoint marker event;

instructions that cause the one or more processors to insert the checkpoint marker event into the continuous input stream of events; and

instructions that cause the one or more processors to generate at least the first batch of events based at least in part on the checkpoint marker event.

20. The non-transitory computer-readable storage memory of claim 17, wherein the plurality of instructions further comprise instructions that cause the one or more processors to trigger generation of a full snapshot of the current state of the system after a pre-determined number of event batches have been processed.