Query processing method for stream data processing systems

• US 7,403,959 B2
• Filed: 08/03/2005
• Issued: 07/22/2008
• Est. Priority Date: 06/03/2005
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Fujiwara, Shinji, Imaki, Tsuneyuki, Nishizawa, Itaru
• Primary Examiner(s): LEROUX, ETIENNE PIERRE
• Application Number:
  US11/195,664
• Publication Number:
  US 20060277230A1
• Time in Patent Office: 1,084 Days
• Field of Search: 707/2, 707/3, 707/4, 707/10, 707/204, 715/500.1, 709/224
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: Y10S 707/99955 Archiving or backup

Abstract

A stream data processing system capable of highly reliable and highly usable real time data processing, is provided with: (1) a mechanism of copying some or all of stream data, and archiving the stream data in a non-volatile memory to allow real time data and archive data to be used seamlessly; and (2) a mechanism of improving the performance of query processings by using a plurality of stream data processing systems cooperatively.

Claims

1. A query processing method for a stream data processing system in which at least one query for performing data process on the stream data is registered and the registered query is executed to process the data stream received from at least one stream data source, the method comprising:

preparing a first query for a real time data processing which stores the received stream data in a buffer and continuously executes the data process on the data in the buffer and a second query for an archive data processing which acquires archive data and executes the data process on the acquired archive data;

archiving some or all of the received stream data into a non-volatile storage medium as the archive data while executing the first query; and

when the stream data cannot be processed by the first query, switching from the first query to the second query to acquire the archive data from the non-volatile storage medium to execute the data process on the acquired archive data.

2. The query processing method for a stream data processing system according to claim 1, wherein when some or all of said received stream data is archived, stream data to be archived is subjected to sampling.

3. The query processing method for a stream data processing system according to claim 1, wherein when some or all of said received stream data is archived, stream data to be archived is subjected to encryption.

4. The query processing method for a stream data processing system according to claim 1, wherein a process of archiving some or all of said received stream data is executed cooperatively with an external data management system.

5. The query processing method according to claim 1, wherein the query is pre-registered with a sliding window specifying a part of the stream data such that a part specified by the sliding window is subjected to the query processing.

6. The query processing method according to claim 1, wherein the query process is read from a program resource via a network and registered.

7. The query processing method according to claim 1, wherein the stream data is acquired from a stream data resource via a network.

8. The query processing method according to claim 1, wherein, when a resource of the stream data processing system is not sufficient to continue to execute the first query, switching to the second query is performed.

9. A query processing method for a stream data processing system in which at least one query for performing data process on the stream data is pre-registered and the pre-registered query is executed to process the data stream received from at least one stream data source, comprising:

archiving some or all of the received stream data into a non-volatile storage medium as archive data while executing the pre-registered query for a real time data processing which stores the received stream data in a buffer and continuously executes the data process on the data in the buffer;

rewriting the pre-registered query to the query for an archive data processing which acquires the archive data from the non-volatile storage medium and executes the data process on the acquired archive data; and

executing the rewritten query to process the archive data.

10. The query processing method according to claim 9, wherein the query is pre-registered with its effective time, and the query is executed during the effective time.

11. The query processing method according to claim 9,

wherein a stream definition defining a format of the stream data subjected to the query processing and a source of the stream data, and an effective term of the data stream is registered when the query is pre-registered, and

wherein the stream data within the effective time is used for the query processing.

12. The query processing method according to claim 9, further comprising:

designating an effective time of archiving operation before the archiving step such that the archiving step is performed during the designated effective time.

13. The query processing method according to claim 9,

wherein a query is processed cooperatively by two or more stream data processing systems, and

wherein the method further comprises:

deciding a process node to be assigned with a query processing in accordance with cost calculation to assign the query processing to the process node in accordance with a decision; and

if query processings required to be performed in order are assigned to different process nodes, generating an intermediate stream interconnecting said process nodes.

14. The query processing method according to claim 13, wherein in accordance with assignment of the process nodes, archive data necessary for the query processing assigned to each process node is copied or transferred to the process node.

15. The query processing method according to claim 9, further comprising:

during the pre-registered query execution, determining whether the buffer for the query processing is sufficient and whether the data is in the buffer;

if the buffer is sufficient and the data is in the buffer, continuing the pre-registered query execution;

if the buffer is not sufficient or the data is not in the buffer, checking whether the archive data in the non-volatile storage medium includes the data necessary to the query processing;

if the archive data includes the data necessary to the query processing, executing the rewritten query; and

combining a result of the pre-registered query execution with a result of the rewritten query execution.