Adding new continuous queries to a data stream management system operating on existing queries

• US 7,996,388 B2
• Filed: 10/17/2007
• Issued: 08/09/2011
• Est. Priority Date: 10/17/2007
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Srinivasan, Anand, Jain, Namit, Mishra, Shailendra Kumar
• Primary Examiner(s): Ali; Mohammad
• Assistant Examiner(s): Bui; Thuy (Tiffany)
• Application Number:
  US11/874,202
• Publication Number:
  US 20090106214A1
• Time in Patent Office: 1,392 Days
• Field of Search: 707/4, 707/E17.014, 707/713, 707/718
• US Class Current: 707/718
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

A new continuous query to a data stream management system (DSMS) may use a stream or a relation which may or may not be used by continuous queries previously registered in the DSMS. The DSMS is programmed to modify an execution plan to accommodate execution of the new query while continuing to execute the previously registered continuous queries. The modified execution plan may include new operators and/or share existing operators. The DSMS is programmed to cause operators which output a relation to propagate a current state of the relation to each newly-coupled operator that uses the relation. The current state is propagated only to operators that have been newly coupled and have thus not yet received any state information previously. After propagation of current state to newly-coupled operators, results of processing any new data for the relation are supplied to all operators coupled thereto, including newly-coupled operators and existing operators.

Claims

1. A computer-implemented method of managing a plurality of streams of data in a computer, the method comprising:

processing the plurality of streams, to execute thereon a plurality of continuous queries based on an existing global plan;

during said processing, receiving a new continuous query to be executed;

during said processing, identifying from the existing global plan, a first operator that supplies data on a relation;

during said processing, modifying the existing global plan by coupling the first operator to a second operator to be used to implement the new continuous query, thereby to obtain a modified global plan;

prior to execution of the new continuous query, a processor in the computer selectively transmitting from the first operator to the second operator but not to a third operator in the existing global plan, a current value of the relation;

prior to receipt of the new continuous query, said current value being transmitted to the third operator from the first operator by use of at least one reference and said at least one reference being deleted after said current value being read by the third operator during said processing based on the existing global plan;

altering said processing, to cause execution of the new continuous query in addition to said plurality of continuous queries, based on the modified global plan; and

outputting from said computer, a stream generated based at least partially on processing of said data by execution of the new continuous query.

2. The method of claim 1 wherein:

said transmitting from the first operator to the second operator is performed at a time when the first operator is awakened subsequent to creation of the modified plan.

3. The method of claim 1 wherein:

each said transmitting is performed in an idempotent manner relative to at least one predetermined soft error.

4. The method of claim 1 further comprising:

initializing at least one parameter of a queue during said modifying; and

changing at least said parameter of said queue during each said transmitting;

wherein said queue supports multiple readers and is coupled between the first operator and each operator in a group of operators in the global plan.

5. The method of claim 1 wherein:

the relation is represented incrementally in the global plan, as a plurality of tuples with each tuple being time stamped; and

said data comprises an initial state and all tuples of the relation having a time stamp older than a current time.

6. The method of claim 1 wherein:

the current state is defined by a sliding window identifying from a stream in the plurality of streams, multiple tuples occurring at a current time instant, in a finite portion of said stream.

7. The method of claim 6 wherein:

the sliding window is a tuple-based window that identifies a fixed number of tuples as specified in said new continuous query.

8. The method of claim 1 wherein:

the relation is represented in a memory coupled to said processor, as an image of the current state, and the image is changed dynamically as the relation changes over time.

9. The method of claim 1 wherein:

after said reference is deleted, said current value still exists in a store; and

said current value is retrieved from said store in response to said transmitting from the first operator to the second operator.

10. The method of claim 1 wherein:

the first operator is a relation operator;

the third operator is an existing operator in the existing global plan;

the second operator is a new operator created during modification of the existing global plan.

11. A non-transitory computer-readable storage medium encoded with instructions for a computer to process streams of data using a plurality of continuous queries in a data stream management system, the instructions comprising:

instructions to execute a global plan for execution of the plurality of continuous queries;

instructions to receive a new continuous query to be executed;

instructions to check if new continuous query comprises a relation already referenced in a continuous query among the plurality of continuous queries being executed as per the global plan;

instructions to instantiate a first operator corresponding to said relation if a result of said checking is false;

instructions to mark, as being available for implementation of said new continuous query, a second operator currently present in the computer and corresponding to said relation, if the result of said checking is true;

instructions to instantiate a third operator corresponding to said new continuous query;

instructions to couple to the third operator, an appropriate operator selected to be one of the first operator and the second operator, wherein selection of the appropriate operator is based on the result of executing said instructions to check;

instructions to transmit to at least the third operator, a current state of the relation, in response to start of execution thereof;

wherein prior to receipt of the new continuous query, said current state comprising data is transmitted from the first operator by use of at least one reference and said at least one reference being deleted after said data in said current state being read by the second operator;

instructions to supply to all operators coupled to the appropriate operator, information related to a new state of the relation, wherein said all operators comprise at least said second operator and said third operator; and

instructions to output from said computer, a stream generated based at least partially on processing of said data by execution of each of said operators.

12. The computer-readable storage medium of claim 11 wherein:

said all operators comprise at least a fourth operator already present in the global plan.

13. The computer-readable storage medium of claim 11 wherein:

said instructions to transmit the current state are to be executed at a time when the appropriate operator is next awakened.

14. The computer-readable storage medium of claim 11 wherein:

said instructions to transmit the current state are executed in an idempotent manner relative to at least one predetermined soft exception.

15. The computer-readable storage medium of claim 11 further comprising:

instructions to initialize at least one parameter of a queue; and

instructions to change at least said parameter of said queue;

wherein said queue supports multiple readers and is coupled between the appropriate operator, the third operator, and a fourth operator already present in the global plan.

16. The computer-readable storage medium of claim 11 wherein:

the current state is defined by a sliding window identifying from a stream in the plurality of streams, multiple tuples occurring at a current time instant, in a finite portion of said stream as specified in said new continuous query.

17. The computer-readable storage medium of claim 16 wherein:

the sliding window is a tuple-based window that identifies a fixed number of tuples.

18. A data stream management system that processes streams of data using a plurality of continuous queries, the data stream management system comprising:

a store encoded with a plurality of tuples representing a relation, each tuple being time stamped;

a memory encoded with a query plan currently being used in execution of the continuous queries;

means, coupled to said store, for modifying the global plan in said memory in response receipt of a new continuous query that uses said relation, if the global plan comprises a first operator that supplies at least one tuple of the relation, by using a multi-reader queue to couple the first operator to a second operator to be used to implement the new continuous query; and

means, coupled to said means for modifying and to said store, for transmitting a current state of the relation, from the first operator to the second operator but not to a third operator, before transmission of a new tuple having a new time stamp;

wherein prior to receipt of the new continuous query, said current state comprising data is transmitted from the first operator by use of at least one reference and said at least one reference being deleted after said data in said current state being read by the third operator.

19. The data stream management system of claim 18 wherein:

said means for transmitting the state is operated at a time when the first operator is awakened subsequent to creation of the modified plan.

20. The data stream management system of claim 18 further comprising:

means for initializing at least one parameter of a queue comprised in said means for modifying; and

means, coupled to said means for transmitting, for changing at least said parameter of said queue;

wherein said queue supports multiple readers and is coupled between the first operator and each operator in a group of operators in the global plan.

21. The data stream management system of claim 18 wherein:

the relation is represented incrementally in the global plan, as a plurality of tuples with each tuple being time stamped; and

an initial state and all tuples of the relation having a time stamp older than a current time represent said current state.