Machine-readable medium for storing a stream data processing program and computer system

• US 8,095,690 B2
• Filed: 10/27/2009
• Issued: 01/10/2012
• Est. Priority Date: 10/28/2008
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Kashiyama, Toshihiko, Nishizawa, Itaru, Imaki, Tsuneyuki
• Primary Examiner(s): Chen, Alan
• Application Number:
  US12/606,641
• Publication Number:
  US 20100106853A1
• Time in Patent Office: 805 Days
• Field of Search: None
• US Class Current: 710/1
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/109 Time management, e.g. calendars, reminders, meetings or time accounting
• CPC Class Codes: G06Q 40/04 Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange

Abstract

In a stream data processing method in which part of stream data is defined as a processing target and the time control information which indicates the advance of the time is inserted into the stream data, time information of the received stream data is stored in a next ignition time holding area as an ignition time. A processing module for generating the time control information at a time different from the time of reception of the stream data is extracted out of a query that indicates how the stream data is to be processed. Based on the extracted processing module and the time information indicating the time of the reception of the stream data, the ignition time is calculated and stored in the next ignition time holding area. The time generation module inserts the time control information at the ignition time held in the next ignition time holding area.

Claims

1. A machine-readable medium for storing a stream data processing program for receiving, as primary information, stream data input to a computer comprising a processor and a storage system, creating secondary information by obtaining a given period of the received primary information as a processing target, and generating time control information indicating advance of time in addition to the primary information,

wherein the stream data processing program controls the computer to execute the procedures of:

receiving the input stream data as the primary information;

calculating, by a time control information generation module for generating, as the time control information, time information at a time of reception of the primary information, a time at which the time control information is generated as a next ignition time, and storing the next ignition time in a next ignition time holding area, which is set in the storage system;

generating the time control information when current time information reaches the next ignition time held in the next ignition time holding area; and

creating, upon reception of the generated time control information, secondary information by obtaining the given period of the received primary information as the processing target.

2. The machine-readable medium for storing a stream data processing program according to claim 1,

wherein the stream data processing program further controls the computer to execute the procedures of:

receiving and registering a query which defines processing specifics for creating the secondary information by obtaining the primary information; and

creating a processing module for executing the processing specifics defined in the received query, and

wherein the procedure of calculating, by the time control information generation module for generating, as the time control information, the time information at the time of the reception of the primary information, the time at which the time control information is generated as the next ignition time, and storing the next ignition time in the next ignition time holding area, which is set in the storage system, comprises the procedures of;

extracting a processing module that needs to generate the time control information at a time different from the time of the reception of the primary information; and

calculating the next ignition time based on the extracted processing module and based on the time information at the time of the reception of the primary information.

3. The machine-readable medium for storing a stream data processing program according to claim 2,

wherein the procedure of calculating, by the time control information generation module for generating, as the time control information, the time information at the time of the reception of the primary information, the time at which the time control information is generated as the next ignition time, and storing the next ignition time in the next ignition time holding area, which is set in the storage system, comprises a procedure of creating a plurality of the time control information generation modules which are connected to the processing module and generate the time control information, and

wherein the procedure of calculating the time at which the time control information is generated as the next ignition time, and storing the next ignition time in the next ignition time holding area, which is set in the storage system, comprises the procedures of:

determining whether or not one of the plurality of the time control information generation modules has received the primary information;

storing the time information of the received primary information in the next ignition time holding area of another one of the plurality of the time control information generation modules which is different from the one of the plurality of the time control information generation modules that has received the primary information; and

storing the time information of the received primary information in a time control information generation-use data processing time holding area for storing the time information processed by the plurality of the time control information generation modules.

4. The machine-readable medium for storing a stream data processing program according to claim 3, wherein the procedure of generating the time control information when the current time information reaches the next ignition time held in the next ignition time holding area comprises the procedures of:

obtaining a current time from a system timestamp holding area, which holds a current time of the computer;

obtaining the next ignition time held in the next ignition time holding area;

determining whether or not the obtained current time is equal to or larger than the next ignition time;

obtaining, when the obtained current time is equal to or larger than the next ignition time, a final data input time from the time control information generation-use data processing time holding area, where a final time at which the primary information has been input to the time control information generation module is held;

determining whether or not the obtained final data input time is smaller than the next ignition time;

updating, when the obtained final data input time is smaller than the next ignition time, the final data input time held in the time control information generation-use data processing time holding area;

generating time control information at the next ignition time; and

deleting, when the obtained final data input time is equal to or larger than the next ignition time, the next ignition time from the next ignition time holding area.

5. The machine-readable medium for storing a stream data processing program according to claim 2,

wherein the procedure of extracting the processing module that needs to generate the time control information at the time different from the time of the reception of the primary information comprises:

a first procedure of determining whether or not the processing module is a processing module that performs, on the primary information, sliding window operation in which a processing target range indicated by a time window that is specified in the query is cut out, and registering, when the processing module is determined as the processing module that performs the sliding window operation, the processing module and a time size in a query execution tree graph analysis result management table, which is used to manage the processing modules that need to generate the time control information at the time different from the time of the reception of the primary information;

a second procedure of determining whether or not the processing module is a processing module that performs, on the primary information, jumping window operation in which a processing target range indicated by a time interval that is specified in the query is cut out, and registering, when the processing module is determined as the processing module that performs the jumping window operation, the processing module and the time interval in the query execution tree graph analysis result management table;

a third procedure of determining whether or not the processing module is a processing module that outputs a query processing result with a delay, and registering, when the processing module is determined as the processing module that outputs the query processing result with the delay, the processing module and a delay size in the query execution tree graph analysis result management table;

a fourth procedure of determining whether or not the processing module is a processing module that outputs a group of the query processing results at regular intervals, and registering, when the processing module is determined as the processing module that outputs the group of the query processing results, the processing module and an output interval in the query execution tree graph analysis result management table; and

a fifth procedure of determining whether or not the processing module is a processing module with a function of excluding, from the query processing results, a processing result that has zero as a length of time in which the processing result remains the processing target, and registering, when the processing module is determined as the processing module with the function of excluding the processing result that has zero, the processing module and a minimum time unit in the query execution tree graph analysis result management table, and

wherein at least one of the first procedure to the fifth procedure is repeated for every processing module in a query execution tree that is created from the registered query.

6. The machine-readable medium for storing a stream data processing program according to claim 5, wherein the procedure of calculating, by the time control information generation module for generating, as the time control information, the time information at the time of the reception of the primary information, the time at which the time control information is generated as the next ignition time, and storing the next ignition time in the next ignition time holding area, which is set in the storage system, comprises the procedures of:

determining whether or not the processing module is the processing module that performs, on the primary information, the sliding window operation in which the processing target range indicated by the time window that is specified in the query is cut out, and registering, when the processing module is determined as the processing module that performs the sliding window operation, in the next ignition time holding area, a value that is obtained by adding input time information and the time size together, the time size being registered in the query execution tree graph analysis result management table;

determining whether or not the processing module is the processing module that performs, on the primary information, the jumping window operation in which the processing target range indicated by the time interval that is specified in the query is cut out, and registering, when the processing module is determined as the processing module that performs the jumping window operation, a value that is obtained by adding time information and the time interval together, the time information indicating a time when the processing module has executed processing, the time interval being registered in the query execution tree graph analysis result management table;

determining whether or not the processing module is the processing module that outputs the query processing result with the delay, and registering, when the processing module is determined as the processing module that outputs the query processing result with the delay, in the next ignition time holding area, a value that is obtained by adding the input time information and the delay size together, the delay size being registered in the query execution tree graph analysis result management table;

determining whether or not the processing module is the processing module that outputs the group of the query processing results at the regular intervals, and registering, when the processing module is determined as the processing module that outputs the group of the query processing results, a value that is obtained by adding the time information and the output interval together, the time information indicating the time when the processing module has executed the processing, the output interval being registered in the query execution tree graph analysis result management table; and

determining whether or not the processing module is the processing module with the function of excluding, from the query processing results, the processing result that has zero as the length of time in which the processing result remains the processing target, and registering, when the processing module is determined as the processing module with the function of excluding the processing result that has zero, in the next ignition time holding area, a value that is obtained by adding the input time information and the minimum time unit together, the minimum time unit being registered in the query execution tree graph analysis result management table.

7. The machine-readable medium for storing a stream data processing program according to claim 5, wherein the procedure of calculating, by the time control information generation module for generating, as the time control information, the time information at the time of the reception of the primary information, the time at which the time control information is generated as the next ignition time, and storing the next ignition time in the next ignition time holding area, which is set in the storage system, comprises the procedures of:

determining for every time control information generation module whether or not the time control information generation module has a parent-child relation in a graph structure that is a tree structure with the extracted processing module; and

storing, when it is determined that the time control information generation module has the parent-child relation, the next ignition time in the next ignition time holding area.

8. The machine-readable medium for storing a stream data processing program according to claim 2,

the computer further comprises:

a first computer; and

a second computer,

the first computer generating the stream data, which is received as the primary information, attaching the time information that is generated in the first computer to the primary information, and transmitting the primary information to the second computer,

the second computer executing processing of creating the secondary information by obtaining the given period of the received primary information as the processing target,

the first computer transmitting the time control information indicating the advance of the time to the second computer in addition to the primary information,

wherein the stream data processing program further controls the second computer to execute the procedures of;

extracting from the query the processing module that needs to generate the time control information at the time different from the time of the reception of the primary information, the query defining the processing specifics for creating the secondary information by obtaining the primary information; and

calculating the next ignition time based on the extracted processing module and based on the time information at the time of the reception of the primary information, and transmitting the next ignition time as a next ignition time message to the first computer, the next ignition time being the time at which the time control information generation module for generating the time control information generates the time control information, and

wherein the stream data processing program further controls the first computer to execute the procedures of;

receiving the next ignition time and storing the received next ignition time in the next ignition time holding area, where the next ignition time is held; and

transmitting the time control information from the time control information generation module to the second computer at the next ignition time held in the next ignition time holding area.

9. The machine-readable medium for storing a stream data processing program according to claim 8, wherein the procedure of transmitting, by the first computer, the time control information from the time control information generation module to the second computer at the next ignition time held in the next ignition time holding area comprises the procedures of:

obtaining a current time from a system timestamp holding area, which holds a current time of the first computer;

obtaining the next ignition time held in the next ignition time holding area;

determining whether or not the obtained current time is equal to or larger than the next ignition time;

generating, when the obtained current time is equal to or larger than the next ignition time, the time control information at the next ignition time; and

deleting, when an obtained final data input time is equal to or larger than the next ignition time, the next ignition time from the next ignition time holding area.

10. The machine-readable medium for storing a stream data processing program according to claim 8,

wherein the procedure of extracting, by the second computer, from the query the processing module that needs to generate the time control information at the time different from the time of the reception of the primary information, the query defining the processing specifics for creating the secondary information by obtaining the primary information comprises:

a sixth procedure of determining whether or not the processing module is a processing module that performs, on the primary information, sliding window operation in which a processing target range indicated by a time window that is specified in the query is cut out, and registering, when the processing module is determined as the processing module that performs the sliding window operation, the processing module and a time size in a query execution tree graph analysis result management table, which is used to manage the processing modules that need to generate the time control information at the time different from the time of the reception of the primary information;

a seventh procedure of determining whether or not the processing module is a processing module that performs, on the primary information, jumping window operation in which a processing target range indicated by a time interval that is specified in the query is cut out, and registering, when the processing module is determined as the processing module that performs the jumping window operation, the processing module and the time interval in the query execution tree graph analysis result management table;

an eighth procedure of determining whether or not the processing module is a processing module that outputs a query processing result with a delay, and registering, when the processing module is determined as the processing module that outputs the query processing result with the delay, the processing module and a delay size in the query execution tree graph analysis result management table;

a ninth procedure of determining whether or not the processing module is a processing module that outputs a group of the query processing results at regular intervals, and registering, when the processing module is determined as the processing module that outputs the group of the query processing results, the processing module and an output interval in the query execution tree graph analysis result management table; and

a tenth procedure of determining whether or not the processing module is a processing module with a function of excluding, from the query processing results, a processing result that has zero as a length of time in which the processing result remains the processing target, and registering, when the processing module is determined as the processing module with the function of excluding the processing result that has zero, the processing module and a minimum time unit in the query execution tree graph analysis result management table, and

wherein at least one of the sixth procedure to the tenth procedure is repeated for every processing module in a query execution tree that is created from the registered query.

11. The machine-readable medium for storing a stream data processing program according to claim 10, wherein the procedure of receiving, by the first computer, the next ignition time and storing the received next ignition time in the next ignition time holding area, where the next ignition time is held comprises the procedures of:

determining whether or not the processing module is the processing module that performs, on the primary information, the sliding window operation in which the processing target range indicated by the time window that is specified in the query is cut out, and registering, when the processing module is determined as the processing module that performs the sliding window operation, in the next ignition time holding area, a value that is obtained by adding input time information and the time size together, the time size being registered in the query execution tree graph analysis result management table;

determining whether or not the processing module is the processing module that performs, on the primary information, the jumping window operation in which the processing target range indicated by the time interval that is specified in the query is cut out, and registering, when the processing module is determined as the processing module that performs the jumping window operation, a value that is obtained by adding time information and the time interval together, the time information indicating a time when the processing module has executed processing, the time interval being registered in the query execution tree graph analysis result management table;

determining whether or not the processing module is the processing module that outputs the query processing result with the delay, and registering, when the processing module is determined as the processing module that outputs the query processing result with the delay, in the next ignition time holding area, a value that is obtained by adding the input time information and the delay size together, the delay size being registered in the query execution tree graph analysis result management table;

determining whether or not the processing module is the processing module that outputs the group of the query processing results at the regular intervals, and registering, when the processing module is determined as the processing module that outputs the group of the query processing results, a value that is obtained by adding the time information and the output interval together, the time information indicating the time when the processing module has executed the processing, the output interval being registered in the query execution tree graph analysis result management table; and

determining whether or not the processing module is the processing module with the function of excluding, from the query processing results, the processing result that has zero as the length of time in which the processing result remains the processing target, and registering, when the processing module is determined as the processing module with the function of excluding the processing result that has zero, in the next ignition time holding area, a value that is obtained by adding the input time information and the minimum time unit together, the minimum time unit being registered in the query execution tree graph analysis result management table.

12. A computer system, comprising:

a processor;

a storage system; and

an interface,

the computer system obtaining, as primary information, stream data which is set in the storage system and input via the interface, creating secondary information by obtaining a given period of the obtained primary information, and generating time control information indicating advance of time in addition to the primary information,

the computer system further comprising;

a query execution tree analyzing module for extracting from a query a processing module that needs to generate the time control information at a time different from a time of reception of the primary information, the query indicating processing specifics for creating the secondary information by obtaining the primary information;

a next ignition time calculating module for calculating an ignition time from based on extracted processing module and based on time information at the time of the reception of the primary information, and storing the ignition time in a next ignition time holding area, which is set in the storage system, the ignition time being a time at which a time control information generation module for generating the time control information generates the time control information; and

a time control information generation module for generating, as the time control information, the time information at the time of the reception of the primary information,

wherein the time control information generation module generates the time control information when current time information reaches the ignition time held in the next ignition time holding area.

13. A computer system, comprising:

a first computer; and

a second computer,

the first computer generating stream data, which is received as primary information, attaching time information that is generated in the first computer to the primary information, and transmitting the primary information to the second computer,

the second computer creating secondary information by obtaining a given period of the received primary information,

the first computer transmitting time control information indicating advance of time to the second computer in addition to the primary information,

the computer system further comprising;

a query execution tree analyzing module of the second computer, for extracting from a query a processing module that needs to generate the time control information at a time different from a time of reception of the primary information, the query indicating processing specifics for creating the secondary information by obtaining the primary information;

a next ignition time calculating module of the second computer, for calculating an ignition time based on the extracted processing module and based on time information at the time of the reception of the primary information, and transmitting the ignition time to the first computer, the ignition time being a time at which a time control information generation module for generating the time control information generates the time control information;

a next ignition time message receiving module of the first computer, for receiving the ignition time and storing the received ignition time in a next ignition time holding area, where the ignition time is held; and

a time control information generation module of the first computer, for transmitting the time control information to the second computer at the ignition time held in the next ignition time holding area.