Stream data anomaly detection method and device

• US 9,305,043 B2
• Filed: 09/12/2011
• Issued: 04/05/2016
• Est. Priority Date: 09/12/2011
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Sakurai, Takao, Hanai, Tomohiro
• Primary Examiner(s): Vital, Pierre
• Assistant Examiner(s): Ho, Andrew N
• Application Number:
  US14/003,563
• Publication Number:
  US 20130346417A1
• Time in Patent Office: 1,667 Days
• Field of Search: 707/741
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/31 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: H04L 1/20 using signal quality detector

Abstract

The present invention is a stream data anomaly detection method for detecting an anomaly in input data received by a computer. The computer has: an input module for receiving the input data; an anomaly detection unit for detecting an anomaly in the input data; and an index management unit for calculating an index for the detection of an anomaly in the input data. The anomaly detection unit compares the received input data with the index and detects an anomaly in the input data when the difference between the value of the input data and the index satisfies a predetermined condition. The index management unit determines whether or not the index should be recalculated, and, when the index is determined to be recalculated, the index management unit recalculates a new index on the basis of the input data and substitutes the index with the calculated index.

Claims

1. A stream data anomaly detection method, for detecting an anomaly from input data received by an interface on a computer,

the computer comprising:

a processor;

a memory;

the interface;

an input module for receiving the input data;

an anomaly detection module for detecting the anomaly from the input data; and

an index management module for managing an index for detecting the anomaly from the input data,

the anomaly detection method for stream data comprising;

a first step of receiving, by the input module, the input data received by the interface;

a second step of comparing, by the anomaly detection module, the received input data with the index, and when a difference between a value of the input data and the index satisfies a predetermined condition, detecting the anomaly from the input data; and

a third step of determining, by the index management module, whether to recalculate the index,

the third step comprising;

a fifth step of calculating a maximum count of the recalculation of the index based on an amount of unprocessed input data that is yet to be processed by the anomaly detection module among pieces of input data received by the input module, an arithmetic operation time required for the recalculation of the index, and a time constraint set in advance;

a sixth step of calculating a predetermined threshold value for each piece of input data received after the previous calculation of the index;

a seventh step of calculating the difference between the value of the input data and the index;

an eighth step of comparing the difference with the predetermined threshold value that corresponds to the piece of input data, determining that the index is to be recalculated when the difference is greater than the predetermined threshold value, and executing processing for executing the recalculation of the index until the maximum count of the recalculation is reached, when the index is determined to be recalculated; and

a ninth step of substituting the index with the recalculated new index.

2. The stream data anomaly detection method according to claim 1, wherein the fifth step comprises setting in advance a processing time required for processing performed by the anomaly detection module for each amount of the unprocessed input data that is yet to be processed by the anomaly detection module, and calculating such a maximum count of the recalculation that a sum of the processing time corresponding to the amount of the unprocessed input data and the arithmetic operation time required for the recalculation of the index falls within the time constraint set in advance.

3. The stream data anomaly detection method according to claim 1, wherein:

the sixth step comprises determining, based on the maximum count of the recalculation, the predetermined threshold value for each of pieces of input data received after the recalculation of the index as a determination criterion for recalculating the index;

the seventh step comprises calculating a distance between the value of the input data and the index as the difference, and calculating a total sum of the distances in order until the number of pieces of unprocessed input data is reached; and

the eighth step comprises comparing the total sum of distances with the predetermined threshold value corresponding to the number of pieces of input data each time the total sum of distances between the value of the input data and the index is calculated in order in the seventh step, and determining that the predetermined threshold value is to be recalculated when the total sum of distances is larger than the predetermined threshold value.

4. The stream data anomaly detection method according to claim 1, further comprising a tenth step of accumulating, by the index management module, a past index when the index is substituted,

wherein the eighth step comprises setting, as the new index, the past index having the difference smaller than that of the current index after executing the processing for executing the recalculation of the index until the maximum count of the recalculation is reached.

5. The stream data anomaly detection method according to claim 1, wherein:

the input module comprises a queue for retaining the input data; and

the third step comprises acquiring the amount of the input data based on a size of the queue containing the unprocessed input data.

6. A stream data anomaly detection device, for detecting an anomaly from input data received by an interface, the anomaly detection device for stream data comprising a computer,

the computer comprising:

a processor;

a memory;

the interface;

an input module for receiving the input data;

an anomaly detection module for detecting the anomaly from the input data; and

an index management module for calculating an index for detecting the anomaly from the input data,

the input module being configured to receive the input data received by the interface,

the anomaly detection module being configured to compare the input data received by the input module with the index, and when a difference between a value of the input data and the index satisfies a predetermined condition, detect the anomaly from the input data,

the index management module comprising;

an index calculation determination module for determining whether to recalculate the index,

the index calculation determination module being configured to;

calculate a maximum count of the recalculation of the index based on an amount of unprocessed input data that is yet to be processed by the anomaly detection module among pieces of input data received by the input module, an arithmetic operation time required for the recalculation of the index, and a time constraint set in advance;

calculate a predetermined threshold value corresponding for each piece of input data received after the previous calculation of the index;

calculate the difference between the value of the input data and the index;

compare the difference with the predetermined threshold value that corresponds to the piece of input data, determine that the index is to be recalculated when the difference is greater than the predetermined threshold value, and execute processing for executing the recalculation of the index until the maximum count of the recalculation is reached, when the index is determined to be recalculated; and

substitute the index with the recalculated new index.

7. The stream data anomaly detection device according to claim 6, wherein the index calculation determination module is further configured to set in advance a processing time required for processing performed by the anomaly detection module for each amount of the unprocessed input data that is yet to be processed by the anomaly detection module, and calculate such a maximum count of the recalculation that a sum of the processing time corresponding to the amount of the unprocessed input data and the arithmetic operation time required for the recalculation of the index falls within the time constraint set in advance.

8. The stream data anomaly detection device according to claim 6, wherein the index calculation determination module is further configured to:

determine, based on the maximum count of the recalculation, the predetermined threshold value for each of the pieces of input data received after the recalculation of the index as a determination criterion for recalculating the index;

calculate a distance between the value of the input data and the index as the difference, and calculate a total sum of the distances in order until the number of pieces of unprocessed input data is reached; and

compare the total sum of distances with the predetermined threshold value corresponding to the number of pieces of input data each time the total sum of distances between the value of the input data and the index is calculated in order, and determine that the predetermined threshold value is to be recalculated when the total sum of distances is larger than the predetermined threshold value.

9. The stream data anomaly detection device according to claim 6, wherein the index calculation module is configured to:

accumulate a past index; and

set, as the new index, the past index having the difference smaller than that of the current index after executing the processing for executing the recalculation of the index until the maximum count of the recalculation is reached.

10. The stream data anomaly detection device according to claim 6, wherein:

the input module comprises a queue for retaining the input data; and

the index calculation determination module is further configured to acquire the amount of the input data based on a size of the queue containing the unprocessed input data.