Stream data processing system and method for avoiding duplication of data process

• US 20070136239A1
• Filed: 11/29/2006
• Published: 06/14/2007
• Est. Priority Date: 12/01/2005
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Kim, Myung-Joon, Lee, Mi-Young, Lee, Hun-Soon
• Granted Patent: US 7,490,078 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2474 Sequence data queries, e.g. querying versioned data
• CPC Class Codes: G06F 16/83 Querying
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation

Abstract

Provided is a stream data processing system and method for avoiding duplication of data process. The system including: an evaluation result storing unit for updating and storing a query condition evaluation result; a window evaluating unit for performing window evaluation; a data separating unit for separating data into new data and duplication input data; a reuse result extracting unit for receiving duplication input data from the data separating unit and extracting a query condition evaluation result; a query condition evaluating unit for receiving new data from the data separating unit, performing query condition evaluation and creating a query condition evaluation result; and a result organizing unit for receiving the query condition evaluation result, merging, outputting and transmitting the query condition evaluation result to the evaluation result storing unit.

Claims

1. A stream data processing system for avoiding duplication of data processing, comprising:

an evaluation result storing means for updating and storing a query condition evaluation result;

a window evaluating means for performing window evaluation on stream data;

a data separating means for separating for stream data transmitted from the window evaluating means into new data and duplication input data based on reusability of the previous query condition evaluation result;

a reuse result extracting means for receiving duplication input data from the data separating means and extracting a query condition evaluation result from the evaluation result storing means;

a query condition evaluating means for receiving new data from the data separating means, performing query condition evaluation and creating a query condition evaluation result; and

a result organizing means for receiving the query condition evaluation results from the reuse result extracting means and the query condition evaluating means, merging, outputting and transmitting the query condition evaluation result to the evaluation result storing means.

2. The system as recited in claim 1, wherein the query condition evaluating means performs continuous query on the new data transmitted from the data separating means.

3. The system as recited in claim 2, wherein the window evaluating means transmits window definition and the window evaluation result to the data separating means when the window evaluation of the stream data is performed.

4. The system as recited in claim 3, wherein when the data separating means separates the stream data based on reusability of the previous query condition evaluation result, the data separating means separates the stream data included in the window evaluation result into duplication input data and new data based on the window definition.

5. The system as recited in claim 4, wherein when the data separating means confirms that there is no pre-stored query condition evaluation result, the data separating means considers the entire data included in the window evaluation result as the new data and transmits the data to the query condition evaluating means.

6. The system as recited in claim 4, wherein the data separating means receives the window definition including window start, window end, tumbling and a window data limiting method from the window evaluating means.

7. The system as recited in claim 6, wherein when the data separating means confirms that there is the pre-stored query condition evaluation result and the window data limiting method is an event, the data separating means considers the data as much as tumbling from the window end as new data, transmits the data to the query condition evaluating means, considers other data as duplication input data, and transmits the other data to the reuse result extracting means.

8. The system as recited in claim 6, wherein when the data separating means confirms that there is the pre-stored query condition evaluation result and the window data limiting method is a time, the data separating means acquires a boundary time by subtracting a tumbling value from the window end time, considers data transmitted before the boundary time as the duplication input data and considers data transmitted after the boundary time as the new data.

9. The system as recited in claim 1, wherein when a query is related to a set function, the result organizing means applies the set function to the merged query condition evaluation result, acquires a final result, outputs the final result with the merged query condition evaluation result.

10. The system as recited in claim 9, wherein the query condition evaluating means performs query condition evaluation on the new data, separates the query evaluation result based on window, document and node levels and hierarchically stores the query evaluation result.

11. The system as recited in claim 10, wherein the window evaluating means performs window evaluation on the stream data of an Extensible Markup Language (XML) form.

12. A method for avoiding duplication of data process in a stream data processing system, comprising the steps of:

a) receiving stream data from outside;

b) performing window evaluation on the stream data;

c) separating the stream data on which window evaluation is performed into duplication input data and new data based on reusability of a previous query condition evaluation result;

d) extracting a query condition evaluation result for the duplication input data from the previous query condition evaluation result;

e) performing query condition evaluation on the new data and creating a query condition evaluation result;

f) merging each query condition evaluation result of the duplication input data and the new data; and

g) updating the previous query condition evaluation result based on the merged query condition evaluation result.

13. The method as recited in claim 12, wherein in the step b), window evaluation is performed on the stream data, and window definition and a window evaluation result are outputted.

14. The method as recited in claim 13, wherein in the step c), when the stream data are separated based on the reusability of the previous query condition evaluation result, the data included in the window evaluation result are separated into duplication input data and new data based on the window definition.

15. The method as recited in claim 14, wherein when it is confirmed that there is no pre-stored query condition evaluation result, entire data included in the window evaluation result are considered as the new data in the step c).

16. The method as recited in claim 14, wherein in the step b), the window definition including window start, window end, tumbling and a window data limiting method is outputted.

17. The method as recited in claim 16, wherein when it is confirmed that there is the pre-stored query condition evaluation result and the window data limiting method is an event, data as much as tumbling from the window end are considered as the new data and other data are considered as the duplication input data in the step c).

18. The method as recited in claim 16, wherein when it is confirmed that there is the pre-stored query condition evaluation result and the window data limiting method is a time, a boundary time is acquired by subtracting a tumbling value from the window end time, and the data transmitted before the boundary time are considered as the duplication input data and the data transmitted after the boundary time are considered as the new data.

19. The method as recited in claim 12, wherein when the query is related to a set function, a final result is outputted by applying the set function to the merged query condition evaluation result in the step f).

20. The method as recited in claim 19, wherein in the step e), the query condition evaluation is performed on the new data and the query evaluation results are hierarchically stored based on window, document and node levels.