STREAM-PROCESSING DATA

• US 20160092317A1
• Filed: 07/15/2015
• Published: 03/31/2016
• Est. Priority Date: 09/29/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Akiyama, Kazuhito, Gotoh, Yasuhisa, Ogihara, Hiroya
• Granted Patent: US 10,146,502 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/1443 Transmit or communication errors
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 7/02 Comparing digital values G06F7/06, G06F7/22, G06F7/38 take precedence
• CPC Class Codes: H04L 1/0084 Formats for payload data

Abstract

A method for stream-processing data including a missing part in real time and thereafter updating the result of the stream processing. A technique for processing data is included. The technique includes receiving data; detecting a probably missing part in the received data while stream-processing the received data in real time; and comparing master data corresponding to the received data and having no missing part with the probably missing part, and if the received data has the missing part, updating the result of the stream processing using the master data.

Claims

1. A method for processing data, the method comprising computer-system implemented steps of:

receiving data;

detecting a probably missing part in the received data while stream-processing of the received data in real time;

comparing master data corresponding to the received data and having no missing part with the probably missing part; and

if the received data has the missing part, updating a result of the stream processing by using the master data.

2. The method according to claim 1, wherein the stream processing of the received data is executed in a first stream processing line that is a combination of a plurality of operators, each operator of the plurality of operators being one processing unit.

3. The method according to claim 2, the method further comprising a computer-system implemented step of:

relocating the combination of the plurality of operators to define a second stream processing line for performing the updating the result of the stream processing.

4. The method according to claim 3, wherein the step of updating the result of the stream processing includes steps of:

processing the master data in accordance with the second stream processing line; and

using a result of the processing the master data to update the result of the stream processing.

5. The method according to claim 3, the method further comprising a computer-system implemented step of:

locating the relocated combination of the plurality of operators on one of a group consisting of a plurality of processor nodes on a computer system and a plurality of virtual processor nodes on the computer system.

6. The method according to claim 3, wherein the relocation of the combination of the plurality of operators is determined in accordance with at least one of a group consisting of a probability of the missing, a type of the missing, a range of influence of the result due to the missing, an order of priority of an output of the processing result, a dependency relationship among operators of the combination of the plurality of operators, a processing time allowable for the updating the result, and a system resource available for the updating the result.

7. The method according to claim 3, wherein if at least one of a group consisting of a processing time allowable for the updating result is limited and a system resource available for the updating the result is limited, the relocating of the combination of the plurality operators is determined in accordance with an order of priority of an output of the result.

8. The method according to claim 1, wherein the step of detecting a probably missing part in the received data further comprises the step of:

outputting at least one of a group consisting of a range of data that may have a missing part, a probability of the missing, a type of the missing, a range of influence of the result due to the missing, an order of priority of an output of the result, a dependency relationship among operators of the combination of the plurality of operators, a processing time required for the operators to perform the stream processing, and a system resource required for the operators to perform the stream processing.

9. The method according to claim 1, wherein the data is measured values from at least one sensor, and wherein the step of detecting the probably missing part in the received data includes the step of:

detecting the probably missing part using the measured values.

10. The method according to claim 1, wherein the data is measured values from at least one sensor, and wherein the step of detecting the probably missing part in the received data further includes the step of:

detecting a collection interval of the measured values.

11. The method according to claim 10, wherein the step of detecting the probably missing part in the received data further includes the step of:

if a difference in the collection interval is detected, detecting data of a part in which the difference is detected as the probably missing part.

12. The method according to claim 11, wherein the step of updating the result of the stream processing includes the step of:

updating, within the result of the stream processing, a second result caused by the missing due to the difference in the collection interval, using a third result obtained from the master data.

13. The method according to claim 1, wherein the step of detecting a probably missing part in the received data occur in response to at least one of a group consisting of the received data having a missing value in a time series, the received data having an abnormal value, the received data having a value that does not change for a fixed period, the received data having a value whose change rate is abnormal, a data collection interval change, a correlation among a plurality of related attributes being abnormal;

a value obtained by comparing differences among values obtained from repeated actions being abnormal, and referencing to missing history data in which a missing history is stored.

14. The method according to claim 1, wherein the step of updating the result of the stream processing includes the step of:

if the received data has the missing part, processing the master data so as to update a result caused from the missing part.

15. The method according to claim 1, wherein the step of updating a result of the stream processing includes the step of:

updating a report configured to indicate an update status in real time.

16. The method according to claim 15, wherein updating the report includes at least one of a group consisting of causing the report to indicate that the update is completed, causing the report to indicate that the data has no missing part, and causing the report to indicate a progress of the update.

17. The method according to claim 1, wherein the step of updating the result of the stream processing includes the steps of:

if the updating cannot be completed within a processing time allowable for the update, stopping the updating process; and

marking an unupdated result in response to stopping the updating process.

18. A computer system for reprocessing data having a missing part, the system comprising at least one computer configured for:

receiving data;

stream-processing the received data in real time;

detecting a probably missing part in the received data during the stream processing;

verifying whether the received data has a missing part by comparing master data corresponding to the received data and having no missing part with the probably missing part; and

updating a result of the stream processing using the master data if the received data has a missing part.

19. The computer system according to claim 18, wherein the at least one computer is further configured for:

executing the stream processing on the received data in a first stream processing line that is a combination of a plurality of operators, each operator of the combination of the plurality of operators being one processing unit; and

relocating the combination of the plurality of operators to define a second stream processing line for performing the updating the result of the stream processing.

20. A non-transitory, computer-readable storage medium having program instructions embodied therewith, the program instructions configured to cause a computer to perform a method comprising:

receiving data;

obtaining a result set by stream processing, in real time, the received data in a first stream processing line;

detecting, while stream-processing of the received data in real time, that a data part is probably not included in the received data;

comparing, in response to the detecting, the received data to a data portion of master data, the master data corresponding to the received data, the data portion including the data part;

determining, based on the comparing, that the data part is not included in the received data;

identifying a first operator of the first stream processing line that is not associated with the data portion;

defining, in response to the determining and based on the identifying, a second processing line by deleting the first operator from the first stream processing line;

obtaining a second result set by stream processing the data portion of the master data in the second processing line; and

updating the first result set using the second result set.