Stream data processing method and system

• US 8,032,554 B2
• Filed: 08/07/2009
• Issued: 10/04/2011
• Est. Priority Date: 10/28/2008
• Status: Active Grant

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Kawashima, Masanori, Handa, Atsuro, Imaki, Tsuneyuki, Nishizawa, Itaru
• Primary Examiner(s): LEWIS, CHERYL RENEA
• Application Number:
  US12/537,676
• Publication Number:
  US 20100106710A1
• Time in Patent Office: 788 Days
• Field of Search: 707/722, 707/725, 715/781, 715/783
• US Class Current: 707/781
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

Stream data is provided for realizing precise calculation processing to keep a constant memory usage of a query including a time-based window, and to take all of input data into consideration. A stream data processing server sections the time-based window into sub-windows each with a smaller width by a query time resolution change unit, and a query processing engine executes the aggregation processing based on the sub-window when receiving the stream data to generate an aggregation tuple, and calculates the calculation result of the query including the time-based window by aggregation processing on the aggregation tuple.

Claims

1. A stream data processing method in a system conducting query processing on stream data, the method comprising:

sectioning, when the query includes a time-based window in which the stream data arriving at the system within a given time interval is to be processed, the time-based window into sub-windows each having a smaller width;

generating at least one aggregation tuple that aggregates the stream data included within the sub-window; and

calculating the query processing result based on the aggregation tuple.

2. The stream data processing method according to claim 1, wherein an aggregation unit of the aggregation tuple is each aggregation group designated by the query.

3. The stream data processing method according to claim 1, wherein the aggregation processing is again executed on the aggregation tuple to calculate the processing result of the query.

4. A stream data processing method in a system conducting query processing on stream data, the method comprising:

sectioning, when the query includes a time-based window in which stream data arriving at the system within a given time interval is to be processed, the time-based window into sub-windows each having a smaller width;

generating, when a received stream tuple is sectioned into the new sub-window, an aggregation tuple within the sub-window;

setting a termination time of a lifetime of the aggregation tuple to a time obtained by adding a width of the time-based window designated by the query to a time stamp of the received stream tuple;

updating the aggregation tuple based on the stream tuple when the received stream tuple is sectioned into the existing sub-window;

setting the lifetime termination time of the updated aggregation tuple to a time obtained by adding a width of the time-based window designated by the query to the time stamp of the received stream tuple;

generating a stream tuple with a flag that erases the aggregation tuple at the set lifetime termination time; and

calculating the query processing result based on the aggregation tuple.

5. The stream data processing method according to claim 4, wherein an aggregation unit of the aggregation tuple is each aggregation group designated by the query.

6. The stream data processing method according to claim 4, wherein the aggregation processing is again executed on the aggregation tuple to calculate the processing result of the query.

7. A stream data processing system for conducting query processing on stream data, comprising:

a network interface unit that receives the stream data;

a processing unit that conducts query processing on the stream data; and

a storage unit that retains the query, wherein the processing unit sections, when the query includes a time-based window in which the stream data arriving at the system within a given time interval is to be processed, the time-based window into sub-windows each having a smaller width, generates at least one aggregation tuple that aggregates the stream data included within the sub-window, and calculates a processing result of the query based on the aggregation tuple.

8. The stream data processing system according to claim 7, wherein an aggregation unit of the aggregation tuple in the processing unit is each aggregation group designated by the query.

9. The stream data processing system according to claim 7, wherein the processing unit again executes the aggregation processing on the aggregation tuple, which is an aggregation result in the sub-window, to calculate the query processing result.

10. The stream data processing system according to claim 7, wherein the processing unit generates, when a received stream tuple is sectioned into a new sub-window, the aggregation tuple within the sub-window;

sets a termination time of a lifetime of the aggregation tuple to a time obtained by adding a width of the time-based window designated by the query to a time stamp of the received stream tuple;

updates the aggregation tuple based on the stream tuple when the received stream tuple is sectioned into the existing sub-window;

sets the lifetime termination time of the updated aggregation tuple to a time obtained by adding a width of the time-based window designated by the query to the time stamp of the received stream tuple; and

generates a stream tuple with a flag that erases the aggregation tuple at the set lifetime termination time.

11. The stream data processing system according to claim 10, wherein an aggregation unit of the aggregation tuple in the processing unit is each aggregation group designated by the query.

12. The stream data processing system according to claim 10, wherein the processing unit again executes the aggregation processing on the aggregation tuple, which is an aggregation result in the sub-window, to calculate the query processing result.

13. The stream data processing system according to claim 7, wherein the processing unit determines possibility for changing the time resolution for determining whether the time-based window can be sectioned into the sub-windows, or not when the query registration processing starts.

14. The stream data processing system according to claim 13, wherein the processing unit determinates possibility for changing the time resolution by determining whether the query processing result is configured by an aggregation function, or not.

15. The stream data processing system according to claim 14, wherein the aggregation function is any one of a summation, a maximum value, a minimum value, a count, and an average value.