Auto-tune anomaly detection

• US 10,600,003 B2
• Filed: 06/30/2018
• Issued: 03/24/2020
• Est. Priority Date: 06/30/2018
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Nie, Kexin, Yang, Yang, Li, Baolei
• Primary Examiner(s): Kim, Matthew M
• Assistant Examiner(s): Putaraksa, Matthew N
• Application Number:
  US16/024,809
• Publication Number:
  US 20200005193A1
• Time in Patent Office: 633 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/0769 Readable error formats, e.g. cross-platform generic formats, human understandable formats
• CPC Class Codes: G06F 11/0772 Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Techniques for auto-tuning anomaly detection are provided. In one technique, training data is stored that comprises training instances, each of which comprises a severity-duration pair and a label that indicates whether the severity-duration pair represents an anomaly. A model is trained based on a first subset of the training data. A second subset of the training data is identified where each training instance includes a positive label that indicates that that training instance represents an anomaly. Based on the second subset of the training data, the model generates multiple scores, each of which corresponds to a different training instance. A minimum score is identified that ensures a particular recall rate of the model. In response to receiving a particular severity-duration pair, the model generates a particular score for the particular severity-duration pair. A notification of an anomaly is generated if the particular score is greater than the minimum score.

Claims

1. A method comprising:

storing training data that comprises a plurality of training instances, each of which comprises a severity-duration pair and a label that indicates whether the severity-duration pair represents an anomaly;

using one or more machine learning techniques to train a model based on a first subset of the training data;

identifying a second subset of the training data, wherein each training instance in the second subset includes a positive label that indicates that said each training instance represents an anomaly;

based on the second subset of the training data, generating, using the model, a plurality of scores, wherein each score corresponds to a different training instance in the second subset;

identifying a minimum score of the plurality of scores that ensures a particular recall rate relative to training instances in the second subset;

in response to receiving a particular severity-duration pair, using the model to generate a particular score for the particular severity-duration pair;

generating a notification of an anomaly if the particular score is greater than the minimum score;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, further comprising:

generating and transmitting, to a remote computing device, performance data that indicates a number of anomalies that the model detected and reported, a response rate, a precision rate of the model, a recall rate of the model, and a user-specified severity-duration pair upon which the model is trained.

3. The method of claim 1, further comprising:

receiving, from a remote computing device, user-provided anomaly data;

based on the user-provided anomaly data, generating a new training instance that comprises a particular severity, a particular duration, and a particular label that indicates whether the particular severity and the particular duration represents an anomaly;

using the one or more machine learning techniques to train an updated model based on the new training instance.

4. The method of claim 3, wherein the user-provided anomaly data includes a timestamp but does not include the particular severity or the particular duration, the method further comprising:

in response to receiving the user-provided anomaly data, determining, based on the timestamp, the particular severity and the particular duration.

5. The method of claim 3, wherein the user-provided anomaly data includes the particular severity and the particular duration.

6. The method of claim 3, further comprising:

generating and transmitting, to a remote computing device, projected performance data that indicates (1) a first performance of the model relative to a set of anomalies and (2) a second performance of the updated model relative to the set of anomalies;

wherein each of the first performance and the second performance indicate a plurality of performance metrics.

7. The method of claim 3, further comprising:

identifying a first set of anomalies that were detected using the model;

identifying a second set of anomalies that would have been detected by the updated model;

identifying a subset of the first set of anomalies that are not found in the second set of anomalies;

transmitting, to a remote computing device, information about the subset of the first set of anomalies.

8. The method of claim 7, wherein the information about the subset of the first set of anomalies includes two or more of a total number of the anomalies in the subset of the first set of anomalies, a number of true anomalies in the subset of the first set of anomalies, a number of false anomalies in the subset of the first set of anomalies, or a number of user-created anomalies in the subset of the first set of anomalies.

9. The method of claim 7, wherein the information includes, for each anomaly in the subset of the first set of anomalies, two or more of the following attributes:

a date of said each anomaly, a duration of said each anomaly, a geographic location of said each anomaly, a severity of said each anomaly, or a resolution of said each anomaly.

10. The method of claim 3, further comprising:

updating the training data to generate updated training data that includes the new training instance and excludes a subset of plurality of training instances;

wherein the updated training data includes one or more training instances of the plurality of training instances;

wherein the updated model is trained based on the updated training data.

11. One or more storage media storing instructions which, when executed by one or more processors, cause:

storing training data that comprises a plurality of training instances, each of which comprises a severity-duration pair and a label that indicates whether the severity-duration pair represents an anomaly;

using one or more machine learning techniques to train a model based on at least a portion of the training data;

in response to receiving a particular severity-duration pair, using the model to generate a particular score for the particular severity-duration pair;

generating a notification of an anomaly if the particular score is greater than a particular threshold;

receiving, from a remote computing device, user-provided anomaly data;

in response to receiving the user-provided anomaly data and based on the user-provided anomaly data, generating a new training instance that comprises a particular severity, a particular duration, and a particular label that indicates whether the particular severity and the particular duration represents an anomaly;

using the one or more machine learning techniques to train an updated model based on the new training instance.

12. The one or more storage media of claim 11, wherein the instructions, when executed by the one or more processors, further cause:

generating and transmitting, to a remote computing device, performance data that indicates a number of anomalies that the model detected and reported, a response rate, a precision rate of the model, a recall rate of the model, and a user-specified severity-duration pair upon which the model is trained.

13. The one or more storage media of claim 11, wherein the user-provided anomaly data includes a timestamp but does not include the particular severity or the particular duration, wherein the instructions, when executed by the one or more processors, further cause:

in response to receiving the user-provided anomaly data, determining, based on the timestamp, the particular severity and the particular duration.

14. The one or more storage media of claim 11, wherein the user-provided anomaly data includes the particular severity and the particular duration.

15. The one or more storage media of claim 11, wherein the instructions, when executed by the one or more processors, further cause:

generating and transmitting, to a remote computing device, projected performance data that indicates (1) a first performance of the model relative to a set of anomalies and (2) a second performance of the updated model relative to the set of anomalies;

wherein each of the first performance and the second performance indicate a plurality of performance metrics.

16. The one or more storage media of claim 11, wherein the instructions, when executed by the one or more processors, further cause:

identifying a first set of anomalies that were detected using the model;

identifying a second set of anomalies that would have been detected by the updated model;

identifying a subset of the first set of anomalies that are not found in the second set of anomalies;

transmitting, to a remote computing device, information about the subset of the first set of anomalies.

17. The one or more storage media of claim 16, wherein the information about the subset of the first set of anomalies includes two or more of a total number of the anomalies in the subset of the first set of anomalies, a number of true anomalies in the subset of the first set of anomalies, a number of false anomalies in the subset of the first set of anomalies, or a number of user-created anomalies in the subset of the first set of anomalies.

18. The one or more storage media of claim 16, wherein the information includes, for each anomaly in the subset of the first set of anomalies, two or more of the following attributes:

a date of said each anomaly, a duration of said each anomaly, a geographic location of said each anomaly, a severity of said each anomaly, or a resolution of said each anomaly.

19. The one or more storage media of claim 11, wherein the instructions, when executed by the one or more processors, further cause:

updating the training data to generate updated training data that includes the new training instance and excludes a subset of plurality of training instances;

wherein the updated training data includes one or more training instances of the plurality of training instances;

wherein the updated model is trained based on the updated training data.

20. The one or more storage media of claim 11, wherein the instructions, when executed by the one or more processors, further cause, prior to receiving the particular severity-duration pair:

identifying a subset of the training data, wherein each training instance in the subset includes a positive label that indicates that said each training instance represents an anomaly;

based on the subset of the training data, generating, using the model, a plurality of scores, wherein each score corresponds to a different training instance in the subset;

identifying a minimum score of the plurality of scores that ensures a particular recall rate relative to training instances in the subset;

wherein the particular threshold is based on the minimum score.