AUTOMATIC ANOMALY DETECTION FRAMEWORK FOR GRID RESOURCES

• US 20170118099A1
• Filed: 10/26/2015
• Published: 04/27/2017
• Est. Priority Date: 10/26/2015
• Status: Active Grant

• Current Assignee: Yahoo Assets LLC
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Huang, Zicheng
• Granted Patent: US 10,454,889 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3065 Monitoring arrangements determined by the means or processing involved in reporting the monitored data error or fault reporting or logging G06F11/0766
• CPC Class Codes: H04L 63/00 Network architectures or network communication protocols for network security cryptographic mechanisms or cryptographic arrangements for secret or secure communication H04L9/00; network architectures or network communication protocols for wireless network security H04W12/00; security arrangements for protecting computers or computer systems against unauthorised activity G06F21/00
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

In one embodiment, a workflow data structure may be generated, updated, or obtained. The workflow data structure may represent system processes, relationships among the system processes, data input to the system processes, data generated by the system processes, and estimated running times associated with the system processes, wherein the data generated by the system processes includes a plurality of metrics. A scheduling map may be generated or updated based, at least in part, on the relationships among the system processes and the estimated running times associated with the system processes, where the scheduling map indicates estimated times at which the metrics are anticipated to be available. The metrics may be monitored based, at least in part, on the scheduling map. Anomalies may be detected according to a result of monitoring the metrics.

Claims

1. A method, comprising:

generating or updating a directed graph representing relationships among a plurality of system processes, wherein edges of the directed graph represent the system processes, values assigned to the edges of the directed graph represent estimated running times associated with the system processes, and nodes of the directed graph include input nodes representing data input to the system processes and output nodes representing data generated by the system processes, wherein the data generated by the system processes include a plurality of metrics;

generating or updating a scheduling map based, at least in part, on the relationships among the system processes and the estimated running times associated with the system processes, wherein the scheduling map indicates estimated times at which the plurality of metrics are anticipated to be available;

monitoring the plurality of metrics based, at least in part, on the scheduling map; and

detecting anomalies according to a result of monitoring the plurality of metrics.

2. The method as recited in claim 1, wherein the directed graph further represents latencies associated with at least a portion of the input nodes, and wherein the scheduling map is generated or updated based, at least in part, on the latencies.

3. The method as recited in claim 1, wherein detecting anomalies in the monitored plurality of metrics comprises detecting an anomaly in a value of one of the plurality of metrics.

4. The method as recited in claim 1, wherein detecting anomalies in the monitored plurality of metrics comprises determining that one of the plurality of metrics is missing.

5. The method as recited in claim 4, further comprising:

waiting a period of time;

raising an alert level;

ascertaining whether the one of the plurality of metrics is available after the period of time; and

transmitting a notification based, at least in part, on the alert level and whether the one of the plurality of metrics is available after the period of time.

6. The method as recited in claim 4, further comprising:

waiting a period of time;

ascertaining whether the one of the plurality of metrics is available after the period of time; and

transmitting a notification based, at least in part, on the period of time and whether the one of the plurality of metrics is available after the period of time.

7. A non-transitory computer-readable storage medium storing thereon computer-readable instructions comprising:

instructions for generating or updating a workflow data structure representing system processes, relationships among the system processes, data input to the system processes, data generated by the system processes, and estimated running times associated with the system processes, wherein the data generated by the system processes includes a plurality of metrics;

instructions for generating or updating a scheduling map based, at least in part, on the relationships among the system processes and the estimated running times associated with the system processes, wherein the scheduling map indicates estimated times at which the plurality of metrics are anticipated to be available;

instructions for monitoring the plurality of metrics based, at least in part, on the scheduling map; and

instructions for detecting anomalies according to a result of monitoring the plurality of metrics.

8. The non-transitory computer-readable storage medium as recited in claim 7, wherein the workflow data structure further represents latencies associated with at least a portion of the data input to the system processes, and wherein the scheduling map is generated or updated based, at least in part, on the latencies.

9. The non-transitory computer-readable storage medium as recited in claim 7, wherein the workflow data structure comprises a directed graph, wherein nodes of the directed graph represent the data input to the system processes and the data generated by the system processes, wherein edges of the directed graph represent the system processes, and wherein values assigned to the edges of the directed graph represent the estimated running times associated with the system processes.

10. The non-transitory computer-readable storage medium as recited in claim 7, wherein detecting anomalies in the monitored plurality of metrics comprises detecting an anomaly in a value of one of the plurality of metrics.

11. The non-transitory computer-readable storage medium as recited in claim 7, wherein detecting anomalies in the monitored plurality of metrics comprises determining that one of the plurality of metrics is missing.

12. The non-transitory computer-readable storage medium as recited in claim 11, further comprising:

instructions for transmitting a notification pertaining to the one of the plurality of metrics.

13. The non-transitory computer-readable storage medium as recited in claim 11, further comprising:

instructions for waiting a period of time;

instructions for raising an alert level;

instructions for ascertaining whether the one of the plurality of metrics is available after the period of time; and

instructions for transmitting a notification based, at least in part, on the alert level and whether the one of the plurality of metrics is available after the period of time.

14. The non-transitory computer-readable storage medium as recited in claim 11, further comprising:

instructions for waiting a period of time;

instructions for ascertaining whether the one of the plurality of metrics is available after the period of time; and

instructions for transmitting a notification based, at least in part, on the period of time and whether the one of the plurality of metrics is available after the period of time.

15. An apparatus, comprising:

a processor; and

a memory, at least one of the processor or the memory being configured to;

obtain, generate, or update a workflow data structure representing system processes, relationships among the system processes, data input to the system processes, data generated by the system processes, and estimated running times associated with the system processes, wherein the data generated by the system processes includes a plurality of metrics;

generate or update a scheduling map based, at least in part, on the relationships among the system processes and the estimated running times associated with the system processes, wherein the scheduling map indicates estimated times at which the plurality of metrics are anticipated to be available;

monitor the plurality of metrics based, at least in part, on the scheduling map; and

detect anomalies according to a result of monitoring the plurality of metrics.

16. The apparatus as recited in claim 15, wherein the workflow data structure further represents latencies associated with at least a portion of the data input to the system processes, and wherein the scheduling map is generated or updated based, at least in part, on the latencies.

17. The apparatus as recited in claim 15, wherein the workflow data structure comprises a directed graph, wherein nodes of the directed graph represent the data input to the system processes and the data generated by the system processes, wherein edges of the directed graph represent the system processes, and wherein values assigned to the edges of the directed graph represent the estimated running times associated with the system processes.

18. The apparatus as recited in claim 15, wherein detecting anomalies in the monitored plurality of metrics comprises detecting an anomaly in a value of one of the plurality of metrics.

19. The apparatus as recited in claim 15, wherein detecting anomalies in the monitored plurality of metrics comprises determining that one of the plurality of metrics is missing.

20. The apparatus as recited in claim 19, wherein at least one of the processor or the memory are further configured to:

wait a period of time;

ascertain whether the one of the plurality of metrics is available after the period of time; and

transmit a notification based, at least in part, on the period of time and whether the one of the plurality of metrics is available after the period of time.