Stream data processing system and stream data processing method

• US 20070022092A1
• Filed: 02/23/2006
• Published: 01/25/2007
• Est. Priority Date: 07/21/2005
• Status: Active Grant

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Imaki, Tsuneyuki, Nishizawa, Itaru
• Granted Patent: US 7,644,110 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: Y10S 715/968 interface for database querying and retrieval

Abstract

A stream data processing system restrains the generation of communication traffic in the case of distributing loads by using a plurality of information processing devices. The global query is divided into a plurality of processes. At every divided process, a local query which includes the corresponding process is generated and each local query is set to a query processing device different from each other. In addition, the stream data is transmitted among a plurality of query processing devices in order so as to execute the local query in each query processing device. At this time, the entire stream data does not transmitted among the query processing devices in order. At least a part of the corresponding stream data is stored in a proxy device. When data received from an information processing device located in immediately before does not include the target data of the local query, the query processing device reads the corresponding data from the proxy device and executes the local query.

Claims

1. A stream data processing system which executes a global query registered beforehand with respect to stream data and outputs an execution result thereof comprising:

i-th (where, 1<

i<

n) query processing devices each of which executes i-th local query composing the global query; and

a proxy device which stores storage data including at least a part of the stream data, wherein

the i-th query processing device includes;

a data receiving unit which receives the stream data in the case of i=1 and receives transfer data from the (i−

1)-th query processing device in the case of i≠

1;

a data registration unit which stores the storage data including at least a part of the received stream data in the proxy device in the case of 1<

i<

n;

a data obtaining unit which accesses the proxy device so as to obtain target data if the storage data includes the target data necessary for processing the i-th local query registered beforehand in the case of 1<

i<

n;

a local query processing unit which executes the i-th local query registered beforehand with respect to the stream data received by the data receiving unit in the case of i=1, and executes an i-th local query registered beforehand with respect to the transfer data received by the data receiving unit and/or the target data obtained by the data obtaining unit in the case of i≠

1; and

a data transmission unit which transmits a processing result of the local query processing unit, as the transfer data, to the (i+1)-th query processing device in the case of i≠

n, and transmits the processing result of the local query processing unit to a predetermined device in the case of i=n, and

the proxy device includes;

a storage data storage unit which receives the storage data from the query processing device and stores the received storage data; and

a data reading unit that reads the target data requested from the query processing device from the storage data stored in the storage data storage unit and transmits the read target data to the query processing device which is a request source.

2. The stream data processing system according to claim 1, wherein

the proxy device further includes:

a global query input unit which receives an input of the global query;

a local query generating unit which generates first to n-th local queries which configure the global query input to the global query input unit; and

a local query transmission unit which transmits the first to n-th local queries generated by the local query generating unit to the first to n-th query processing devices, and

the i-th (where, 1<

i<

n) query processing device further includes;

a query receiving unit which receives the i-th local query from the proxy device and sets the received i-th local query to the local query processing unit.

3. The stream data processing system according to claim 2, wherein

the local query generating device generates the first to n-th local queries so as for the total sum of a processing cost to satisfy a predetermined condition when executing the first to n-th local queries in the local query processing unit of the first to n-th query processing devices, respectively.

4. The stream data processing system according to claim 3, wherein

the local query generating unit includes:

a generating unit which generates a plurality of groups of the first to n-th local queries with respect to the global query input to the global query input unit;

an execution unit which executes, for each of the first to n-th query processing devices, a process that transmits the i-th (where, 1<

i<

n) local query of the each generated group to the i-th query processing device, and obtains a processing cost of the i-th local query of the each generated group from the query processing device concerned; and

a selection unit which selects a group whose total sum of the processing cost of the first to n-th local queries is minimized, on the basis of the processing cost of the first to n-th local queries of the each group obtained from the first to n-th query processing devices, and wherein

the local query transmission unit transmits the first to n-th local queries of the group selected by the selection unit to the first to n-th query processing device, respectively.

5. The stream data processing system according to claim 2,wherein

the proxy device further includes a storage data requesting unit which transmits, to the i-th (where, 1<

i<

n) query processing device, an assignment of a part, which is to be stored as the storage data in the data storage unit, among the stream data input to the i-th query processing device concerned, and

the data registration unit extracts the part assigned by the proxy device from the stream data received by the data receiving unit and stores storage data which includes the extracted part concerned.

6. The stream data processing system according to claim 5, wherein

the proxy device further includes a storage data determining unit which determines a part, which is to be stored as the storage data in the data storage unit, among the stream data so that the total sum of communication costs for obtaining the target data, which is necessary to execute the i-th (where, 1<

i<

n) local query generated by the local query generating unit in the i-th query processing device, from the stream data concerned input to a j-th (where, 1<

j<

i) query processing device, can satisfy the predetermined condition.

7. The stream data processing system according to claim 6, wherein

the storage data determining unit executes, for the first to n-th query processing devices, a process which compares a first communication cost when obtaining the target data, in which the i-th (where, 2<

i<

n) query processing device needs so as to execute the i-th local query, from the stream data through the j-th (where, 1<

j<

i) query processing device with a second communication cost when obtaining the target data from the stream data through the proxy device, and determines the target data, as the storage data which is a part to be stored in the data storage unit when the second communication cost is less than the first communication cost.

8. The stream data processing system according to claim 1, wherein

the proxy device further includes a system time stamp generating unit which generates a system time stamp, which indicates an input order of the stream data input to the first query processing device, in response to a system time stamp request from the first query processing device, and transmits the generated system time stamp to the first query processing device concerned,

the data receiving unit of the first query processing device transmits the system time stamp request to the proxy device whenever newly receiving stream data, obtains the system time stamp from the proxy device, and assigns the obtained system time to newly received stream data,

the local query processing unit of the i-th (where, 1<

i<

n) query processing device executes the i-th local query with respect to a part of the corresponding stream data according to the order indicated by the system time stamp assigned to the stream data in the case of i=1, and executes the i-th local query with respect to the corresponding transmission data and/or the target data according to the order indicated by the system time stamp assigned to the transmission data in the case of i≠

1, and

the data transmission unit of the i-th (where, 1<

i<

n) query processing device assigns the system time stamp assigned to the stream data received by the data receiving unit to the transmission data which is the processing result of the local query processing unit and transmits the transmission data to the second query processing device in the case of i=1, and assigns the system time stamp assigned to the transmission data received by the data receiving unit to the transmission data which is the processing result of the local query processing unit and transmits the transmission data to an (i+1)-th query processing device in the case of i≠

1,n.

9. The stream data processing system according to claim 8, wherein

the proxy device further includes an expire time stamp generating unit which generates an expire time stamp which indicates a lifetime of the stream data, in which the system time stamp generated by the system time stamp generating unit is assigned, in the stream data processing system, and

the i-th (where, 1<

i<

n) query processing device further includes;

an obtaining unit which obtains an expire time stamp, which corresponds to the system time stamp assigned to the stream data received by the data receiving unit, from the proxy device in the case of i=1; and

a delete unit which deletes corresponding stream data which exists in the corresponding query processing device and includes an expire time stamp older than the system time stamp of the stream data determined as a current process target in the corresponding query processing device in the case of 2<

i<

n.

10. The stream data processing system according to claim 1, wherein

the data storage unit includes a main storage device and an auxiliary storage device.

11. The stream data processing system according to claim 1, wherein

the predetermined device is the proxy device.

12. A proxy device used in the stream data processing system according to claim 1.

13. A query processing device used in the stream data processing system according to claim 1.

14. A stream data processing method of causing a stream data processing system, which includes i-th (where, 1<

i<

n) query processing devices each of which executes an i-th local query composing the global query and a proxy device which stores storage data including at least a part of the stream data, to perform;

executing a global query with respect to stream data; and

outputting an execution result thereof,

wherein the i-th query processing device performs the steps of;

receiving the stream data in the case of i=1 and receiving transfer data from an (i−

1)-th query processing device in the case of i≠

1;

transmitting storage data including at least a part of the received stream data to the proxy device in the case of 1<

i<

n;

accessing the proxy device so as to obtain target data if the storage data includes the target data necessary for processing the i-th local query registered beforehand in the case of 1<

i<

n;

executing the i-th local query registered beforehand with respect to the received stream data in the case of i=1, or executing an i-th local query registered beforehand with respect to the received transfer data and the obtained target data in the case of i≠

1; and

transmitting an execution result of the local query, as transfer data, to an (i+1)-th query processing device in the case of i≠

n, and transmitting the execution result of the local query to a predetermined device in the case of i=n, and

wherein the proxy device performs the steps of;

receiving the storage data from the query processing device and storing the received storage data; and

reading target data requested by the query processing device from the stored storage data and transmitting the read target data to the query processing device which is a request source.