Managing snapshots and application state in micro-batch based event processing systems

• US 10,713,249 B2
• Filed: 09/15/2017
• Issued: 07/14/2020
• Est. Priority Date: 09/15/2016
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Park, Hoyong, Bishnoi, Sandeep, Thukkaram, Prabhu, Kumar, Santosh, Advani, Pavan, Mulay, Kunal, Toillion, Jeffrey
• Primary Examiner(s): Peng, Huawen A
• Application Number:
  US15/706,329
• Publication Number:
  US 20180075046A1
• Time in Patent Office: 1,033 Days
• CPC Class Codes: G06F 11/3003 Monitoring arrangements specially adapted to the computing system or computing system component being monitored
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 16/9032 Query formulation
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/90344 by using string matching techniques

Abstract

An event processing system for processing events in an event stream is disclosed. The system can execute instructions to receive a micro-batch stream of input events, process the input events using the CQL engine to generate a set of output events, generate, using a snapshot management algorithm implemented by the CQL engine, a snapshot of a current state of a system based at least in part on the set of output events related to the application, generate a first directory structure to access snapshot information associated with the snapshot of the current state of the system, generate a second directory structure to generate a list of snapshots associated with the current state of the system, and determine based at least in part on the snapshot management algorithm, a process to get, add, or clean the list of snapshots associated with the current state of the system.

Claims

1. A method for managing snapshots created from a Continuous Query Language (CQL) engine, comprising:

receiving, by a computing device, a continuous query;

applying, by the computing device, a Directed Acrylic Graph (DAG) transformation to the continuous query to generate a query plan for the continuous query, wherein the query plan is an ordered set of steps used to access data for processing of the continuous query;

applying, by the computing device, a CQL transformation to the query plan to generate a transformed query plan;

receiving, by a computing device, a micro-batch stream of input events related to an application;

processing, by the computing device, the input events using the CQL engine to generate a set of output events related to the application, wherein the processing comprises;

performing, by the CQL engine, incremental computation on each of the input events of the micro-batch stream for the continuous query based at least in part on the transformed query plan; and

creating, by the CQL engine, output events for each of the input events of the micro-batch stream, wherein the set of output events comprise the output events for each of the input events of the micro-batch stream;

generating, by the computing device and using a snapshot management algorithm implemented by the CQL engine, a snapshot of a current state of a system based at least in part on the set of output events related to the application;

generating, by the computing device, a first directory structure to access snapshot information associated with the snapshot of the current state of the system;

generating, by the computing device, a second directory structure to generate a list of snapshots associated with the current state of the system; and

determining, by the computing device, based at least in part on the snapshot management algorithm, a process to get, add, or clean the list of snapshots associated with the current state of the system.

2. The method of claim 1, wherein the micro-batch stream is a continuous stream of data discretize into sub-second micro-batches.

3. The method of claim 1, further comprising storing, by the computing device, the set of output events related to the application in an output queue;

and transmitting, by the computing device, the output events in the output queue when all of the input events have been processed.

4. The method of claim 3, wherein the micro-batch stream comprises micro-batches of data or Resilient Distributed Datasets (RDDs).

5. The method of claim 4, wherein the processing each of the input events comprises performing a computation on each of the input based at least in part on the transformed query plan.

6. The method of claim 5, wherein the continuous query includes pattern matching.

7. A system, comprising:

a memory configured to store computer-executable instructions; and

a processor configured to access the memory and execute the computer-executable instructions to;

receive a continuous query;

apply a Directed Acrylic Graph (DAG) transformation to the continuous query to generate a query plan for the continuous query, wherein the query plan is an ordered set of steps used to access data for processing of the continuous query;

apply a Continuous Query Language (CQL) transformation to the query plan to generate a transformed query plan such that a CQL engine can execute the continuous query using the transformed query plan;

receive a micro-batch stream of input events related to an application;

process the input events using the CQL engine to generate a set of output events related to the application, wherein the processing comprises;

performing, by the CQL engine, incremental computation on each of the input events of the micro-batch stream for the continuous query based at least in part on the transformed query plan; and

creating, by the CQL engine, output events for each of the input events of the micro-batch stream, wherein the set of output events comprise the output events for each of the input events of the micro-batch stream;

generate, using a snapshot management algorithm implemented by the CQL engine, a snapshot of a current state of a system based at least in part on the set of output events related to the application;

generate a first directory structure to access snapshot information associated with the snapshot of the current state of the system;

generate a second directory structure to generate a list of snapshots associated with the current state of the system; and

determine based at least in part on the snapshot management algorithm, a process to get, add, or clean the list of snapshots associated with the current state of the system.

8. The system of claim 7, wherein the micro-batch stream is a continuous stream of data discretize into sub-second micro-batches.

9. The system of claim 7, wherein the computer executable instructions are further executable to store the set of output events related to the application in an output queue;

and transmit the output events in the output queue when all of the input events have been processed.

10. The system of claim 9, wherein the micro-batch stream comprises micro-batches of data or Resilient Distributed Datasets (RDDs).

11. The system of claim 10, wherein the processing each of the input events comprises performing a computation on each of the input based at least in part on the transformed query plan.

12. The system of claim 11, wherein wherein the continuous query includes pattern matching.

13. A computer-readable medium storing computer-executable code that, when executed by a processor, cause the processor to perform operations comprising:

receiving a continuous query;

applying a Directed Acrylic Graph (DAG) transformation to the continuous query to generate a query plan for the continuous query, wherein the query plan is an ordered set of steps used to access data for processing of the continuous query;

applying a Continuous Query Language (CQL) transformation to the query plan to generate a transformed query plan such that a CQL engine can execute the continuous query using the transformed query plan;

receiving a micro-batch stream of input events related to an application;

processing the input events based at least in part on the transformed query plan using the CQL engine to generate a set of output events related to the application, wherein the processing comprises;

performing, by the CQL engine, incremental computation on each of the input events of the micro-batch stream for the continuous query based at least in part on the transformed query plan; and

creating, by the CQL engine, output events for each of the input events of the micro-batch stream, wherein the set of output events comprise the output events for each of the input events of the micro-batch stream;

generating, using a snapshot management algorithm implemented by the CQL engine, a snapshot of a current state of a system based at least in part on the set of output events related to the application;

generating a first directory structure to access snapshot information associated with the snapshot of the current state of the system;

generating a second directory structure to generate a list of snapshots associated with the current state of the system; and

determining based at least in part on the snapshot management algorithm, a process to get, add, or clean the list of snapshots associated with the current state of the system.

14. The computer-readable medium of claim 13, wherein the micro-batch stream is a continuous stream of data discretize into sub-second micro-batches.

15. The computer-readable medium of claim 13, wherein operations further comprise storing the set of output events related to the application in an output queue;

and transmitting the output events in the output queue when all of the input events have been processed.

16. The computer-readable medium of claim 15, wherein the micro-batch stream comprises micro-batches of data or Resilient Distributed Datasets (RDDs).

17. The computer-readable medium of claim 16, wherein the processing each of the input events comprises performing a computation on each of the input based at least in part on the transformed query plan.