STREAM DATA PROCESSING METHOD AND APPARATUS

• US 20110040827A1
• Filed: 02/19/2010
• Published: 02/17/2011
• Est. Priority Date: 08/12/2009
• Status: Active Grant

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Katsunuma, Satoshi, Fujiwara, Shinji, Imaki, Tsuneyuki
• Granted Patent: US 8,176,196 B2
• Time in Patent Office: Days
• US Class Current: 709/203
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

A user specifies a group by a user-defined query. An input order is assured among data of a same group. By outputting data belonging to a same group from the same node, an order inputted for a necessary portion is assured. By outputting data belonging to different groups from another node, processing in plural nodes is achieved to avoid reduction in performance.

Claims

1. A stream data processing method comprising the steps of:

registering a query in a first node, subjecting data added with time stamps to operator execution in a second node set, and sorting the data having been subjected to operator execution according to the time stamps to output the data in a third node set;

extracting, in the first node, a set of first value expressions indicating a group that assures an input order by data of the same group from the registered query, and calculating a set of second value expressions indicating a method for partitioning operator execution for each data in the plural second nodes;

calculating a set of third value expressions indicating a method for performing output processing by partitioning data into the plural third nodes from the sets of the first and second value expressions, and determining a method for distributing data to the third node set from the set of third value expressions; and

determining, on the basis of the distribution method for the third node set and the set of the first value expressions, a method for distributing data having been processed in the same second node so as to be processed in the same third node.

2. The stream data processing method according to claim 1, wherein the calculating the set of third value expressions comprises of the step of:

defining, when any value expression F exists as an element in the sets of the first and second value expressions, the value expression F as an element in the set of third value expressions.

3. The stream data processing method according to claim 1, wherein the calculating the set of third value expressions comprises of the step of:

defining, for any value expression F as an element of the set of first value expressions, and any value expression G as an element of the set of second value expressions, when it is shown that stream data having a same value as the value expression F always has a same value as the value expression G, an element in the set of third value expressions as the value expression G.

4. The stream data processing method according to claim 1, wherein the determining the method for distributing data to the third node set determines the method for distributing data to the third node set from the set of the third value expressions, the number of the second nodes, the number of the third nodes, and information about stream data;

and

wherein the method further comprising the step of analyzing and determining a method for distributing data of the second node set like the distribution method of the third node set, or a method for distributing data to more nodes than the third node set, on the basis of the distribution method for the third node set, the set of the first value expressions, the number of the second nodes, and information about stream data.

5. The stream data processing method according to claim 1, further comprising the steps of:

retrieving, in the first node or a client node, the third node set with referring to information about the data distribution method of the third node set;

sending data and information about the client to the third node set; and

transferring, by the third node set, output data to the client, based on the data and the information about the client.

6. The stream data processing method according to claim 1, further comprising the step of:

adjusting, in the second node set or the third node set, output time for data to be outputted by using time stamp values of the data to be outputted and a permissible time lag width to confine the deviation of the order of inputted data within an indicated range when the permissible time lag width of an input order among the data to be outputted is presented from a client node.

7. The stream data processing method according to claim 1, further comprising the step of distributing data from the first node to plural time stamp adding nodes so that data to be processed in the same third node are subjected to time stamp adding processing in the same time stamp adding node.

8. A stream data processing system that processes inputted data comprising:

a first node for registering a query;

a second node set for performing operator execution of data to which a time stamp is added; and

a third node set for sorting the data having been subjected to said operator execution according to the time stamp, and for outputting the data,

wherein the first node extracts a set of first value expressions indicating a group that assures an input order by data of the same group from the registered query, and calculates a set of second value expressions indicating a method for partitioning operator execution for each data in the plural second nodes,

wherein the first node calculates a set of third value expressions indicating a method for performing output processing by partitioning data into the plural third nodes from the sets of the first and second value expressions, and generates a first table indicating a method for distributing data to the third node set from the set of third value expressions, and

wherein the first node generates a second table indicating a method for distributing data having been processed in the same second node so as to be processed in the same third node, from the distribution method for the third node set and the set of the first value expressions.

9. The stream data processing system according to claim 8, wherein the first node defines, when any value expression F exists as an element in the sets of the first and second value expressions, the value expression F as an element in the set of third value expressions.

10. The stream data processing system according to claim 8, Wherein the first node defines, for any value expression F as an element of the set of first value expressions, and any value expression G as an element of the set of second value expressions, when it is shown that stream data having the same value as the value expression F always has the same value as the value expression G, an element in the set of third value expressions as the value expression G.

11. The stream data processing system according to claim 8, wherein the first node generates the first table indicating a method for distributing data to the third node set from the set of the third value expression, the number of the second nodes, the number of the third nodes, and information about stream data,

wherein the first node determines, from the distribution method in the third node set, the set of the first value expressions, the number of the second nodes, and information about stream data, a method for distributing data of the second node set like the distribution method of the third node set, or a method for distributing data to more nodes than the third node set, and

wherein the first node generates the second table indicating the data distribution method of the second node set from the determined distribution method of the second node set.

12. The stream data processing system according to claim 8,

wherein the first node retrieves the third node set from the first table, and sends data and information about a client node to the third node set, and

wherein the third node set transfers data to the client node, based on the acquired information about the client node.

13. The stream data processing system according to claim 11, wherein the second node set or the third node set adjusts, when a permissible time lag width of an input order among data to be outputted is presented from a client node, output time of the data to be outputted by using time stamp values of the data to be outputted and the permissible time lag width to confine the deviation of the order of inputted data within an indicated range.

14. The stream data processing system according to claim 8, further comprising plural time stamp adding nodes, and

Wherein the first node generates a table indicating a method for distributing the data to the plural time stamp adding nodes, the method being that data processed in the same third node to be processed in the same third node are subjected to time stamp adding processing in the same time stamp adding node.