Real-time adaptive operations performance management system using event clusters and trained models

• US 9,582,781 B1
• Filed: 09/01/2016
• Issued: 02/28/2017
• Est. Priority Date: 09/01/2016
• Status: Active Grant

• Current Assignee: Pagerduty, Inc.
• Original Assignee: Pagerduty, Inc.
• Inventors: Kearns, Justin David, Zuchlewski, Laura Ann, Ronen, Ophir
• Primary Examiner(s): Vincent, David
• Application Number:
  US15/254,996
• Time in Patent Office: 180 Days
• Field of Search: 706/12, 706/45
• US Class Current: 1/1
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/0637 Strategic management or analysis, e.g. setting a goal or target of an organisation; Planning actions based on goals; Analysis or evaluation of effectiveness of goals

Abstract

Embodiments are directed to managing operations. If Operations events are provided, event clusters may be associated with one or more Operations events, such that the Operations events may be associated with the event clusters based on characteristics of the Operations events. Metrics including resolution metrics, root cause analysis, notes, and other remediation information may be associated with the event clusters. Then a modeling engine may be employed to train models based on the Operations events, the event clusters, and the resolution metrics, such that the trained model may be trained to correlate and predict the resolution metrics from real-time Operations events. If real-time Operations events may be provided, the trained models may be employed to predict the resolution metrics that are associated with the real-time Operations events. If model performance degrades beyond accuracy requirements, new observations may be added to the training set and the model re-trained.

Claims

1. A method for managing operations for organizations over a network using one or more network computers that include one or more processors that perform actions, comprising:

when a plurality of Operations events are provided, performing further actions, including;

providing, by the one or more processors, one or more event clusters that are associated with one or more Operations events of the plurality of Operations events, wherein the one or more Operations events are associated with the one or more event clusters based on one or more characteristics of the one or more Operations events;

associating, by the one or more processors, one or more resolution metrics with the one or more event clusters, wherein the association is based on one or more of human-validated interface applications or generalized models;

employing, by the one or more processors, a modeling engine to train one or more models based on the one or more Operations events, the one or more event clusters, and the one or more resolution metrics, wherein the trained model is configured to correlate and predict the one or more resolution metrics from one or more real-time Operations events;

configuring and arranging, by the one or more processors, a non-transitory computer readable media for storing the one or more trained models;

storing, by the one or more processors, the one or more trained models in the non-transitory computer readable media; and

when the one or more real-time Operations events are provided, performing further actions including;

retrieving, by the one or more processors, the one or more trained models from the non-transitory computer readable memory; and

employing, by the one or more processors, the one or more trained models to identify the one or more resolution metrics that are associated with the one or more real-time Operations events.

2. The method of claim 1, further comprising, transforming, by the one or more processors, each Operations event in the plurality of Operations events into a common event format.

3. The method of claim 1, further comprising, reducing, by the one or more processors, a number of Operations events included in the plurality of Operations events based on one or more of, source origin identifier value, source component identifier value, or service identifier value.

4. The method of claim 1, further comprising, when model performance degrades beyond accuracy requirements as measured by error produced in the output predictions, re-training, by the one or more processors, the trained model based in part on one or more other Operations events, one or more other event clusters, and one or more other resolution metrics.

5. The method of claim 1, wherein associating, by the one or more processors, the one or more resolution metrics with the one or more event clusters, further comprises, associating one or more of, a time-to-resolve value, a number of responders required to resolve, one or more names of responders or teams required to resolve, or a priority of the incident in terms of operations and business impact.

6. The method of claim 1, wherein providing the plurality of Operations events, further comprises, providing one or more portions of the plurality of Operations events, wherein the one or more portions of Operations events are provided by two or more separate but related sources or services.

7. The method of claim 1, wherein training the one or more models further comprises, employing modeling techniques, including one or more of curve-fitting, regression, statistical modeling, or machine learning to identify one or more features of the one or more event clusters that are incorporated into the one or more trained models.

8. The method of claim 1, wherein training the one or more models further comprises, employing modeling techniques including one or more of curve-fitting, regression, statistical modeling, or machine learning to correlate and predict resolution metrics given one or more event cluster features.

9. A system for managing operations for organizations over a network, comprising:

a network computer, comprising;

a transceiver that communicates over the network;

a memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including;

when a plurality of Operations events are provided, performing further actions, including;

providing one or more event clusters that are associated with one or more Operations events of the plurality of Operations events, wherein the one or more Operations events are associated with the one or more event clusters based on one or more characteristics of the one or more Operations events;

associating one or more resolution metrics with the one or more event clusters, wherein the association is based on one or more of human-validated interface applications or generalized models;

employing a modeling engine to train one or more models based on the one or more Operations events, the one or more event clusters, and the one or more resolution metrics, wherein the trained model is configured to correlate and predict the one or more resolution metrics from one or more real-time Operations events;

configuring and arranging the memory for storing the one or more trained models;

storing the one or more trained models in the memory;

when the one or more real-time Operations events are provided, performing further actions including;

retrieving the one or more trained models from the memory; and

employing the one or more trained models to identify the one or more resolution metrics that are associated with the one or more real-time Operations events; and

a client computer, comprising;

a client computer transceiver that communicates over the network;

a client computer memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including;

displaying the plurality of Operations events;

obtaining user inputs for associating the one or more resolution metrics with the one or more event clusters; and

communicating the user inputs to the network computer.

10. The system of claim 9, wherein the one or more network computer processors execute instructions that perform actions, further comprising, transforming each Operations event in the plurality of Operations events into a common event format.

11. The system of claim 9, wherein the one or more network computer processors execute instructions that perform actions, further comprising, reducing a number of Operations events included in the plurality of Operations events based on one or more of, source origin identifier value, source component identifier value, or service identifier value.

12. The system of claim 9, wherein the one or more network computer processors execute instructions that perform actions, further comprising, when model performance degrades beyond accuracy requirements as measured by error produced in the output predictions, re-training the trained model based on one or more other Operations events, one or more other event clusters, and one or more other resolution metrics.

13. The system of claim 9, wherein associating, by the one or more processors, the one or more resolution metrics with the one or more event clusters, further comprises, associating one or more of, a time-to-resolve value, a number of responders required to resolve, one or more names of responders or teams required to resolve, or a priority of the incident in terms of operations and business impact.

14. The system of claim 9, wherein providing the plurality of Operations events, further comprises, providing one or more portions of the plurality of Operations events, wherein the one or more portions of Operations events are provided by two or more separate but related sources or services.

15. The system of claim 9, wherein training the one or more models further comprises, employing modeling techniques, including one or more of curve-fitting, regression, statistical modeling, or machine learning to identify one or more features of the one or more event clusters that are incorporated into the one or more trained models.

16. The system of claim 9, wherein training the one or more models further comprises, employing modeling techniques including one or more of curve-fitting, regression, statistical modeling, or machine learning to correlate and predict resolution metrics given one or more event cluster features.

17. A processor readable non-transitory storage media that includes instructions for managing operations for organizations over a network, wherein execution of the instructions by one or more hardware processors performs actions, comprising:

when a plurality of Operations events are provided, performing further actions, including;

providing one or more event clusters that are associated with one or more Operations events of the plurality of Operations events, wherein the one or more Operations events are associated with the one or more event clusters based on one or more characteristics of the one or more Operations events;

associating one or more resolution metrics with the one or more event clusters, wherein the association is based on one or more of human-validated interface applications or generalized models;

employing a modeling engine to train one or more models based on the one or more Operations events, the one or more event clusters, and the one or more resolution metrics, wherein the trained model is configured to correlate and predict the one or more resolution metrics from one or more real-time Operations events;

configuring and arranging another non-transitory computer readable media for storing the one or more trained models;

storing the one or more trained models in the other non-transitory computer readable media;

when the one or more real-time Operations events are provided, performing actions including;

retrieving the one or more trained models from the other non-transitory computer readable memory; and

employing the one or more trained models to identify the one or more resolution metrics that are associated with the one or more real-time Operations events.

18. The media of claim 17, further comprising, transforming, by the one or more processors, each Operations event in the plurality of Operations events into a common event format.

19. The media of claim 17, further comprising, reducing, by the one or more processors, a number of Operations events included in the plurality of Operations events based on one or more of, source origin identifier value, source component identifier value, or service identifier value.

20. The media of claim 17, further comprising, when model performance degrades beyond accuracy requirements as measured by error produced in the output predictions, re-training, by the one or more processors, the trained model based on one or more other Operations events, one or more other event clusters, and one or more other resolution metrics.

21. The media of claim 17, wherein associating, by the one or more processors, the one or more resolution metrics with the one or more event clusters, further comprises, associating one or more of, a time-to-resolve value, a number of responders required to resolve, one or more names of responders or teams required to resolve, or a priority of the incident in terms of operations and business impact.

22. The media of claim 17, wherein providing the plurality of Operations events, further comprises, providing one or more portions of the plurality of Operations events, wherein the one or more portions of Operations events are provided by two or more separate but related sources or services.

23. The media of claim 17, wherein training the one or more models further comprises, employing modeling techniques, including one or more of curve-fitting, regression, statistical modeling, or machine learning to identify one or more features of the one or more event clusters that are incorporated into the one or more trained models.

24. A network computer for managing operations for organizations over a network, comprising:

a transceiver that communicates over the network;

a memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including;

when a plurality of Operations events are provided, performing further actions, including;

providing one or more event clusters that are associated with one or more Operations events of the plurality of Operations events, wherein the one or more Operations events are associated with the one or more event clusters based on one or more characteristics of the one or more Operations events;

associating one or more resolution metrics with the one or more event clusters, wherein the association is based on one or more of human-validated interface applications or generalized models;

employing a modeling engine to train one or more models based on the one or more Operations events, the one or more event clusters, and the one or more resolution metrics, wherein the trained model is configured to correlate and predict the one or more resolution metrics from one or more real-time Operations events;

configuring and arranging the memory for storing the one or more trained models;

storing the one or more trained models in the memory;

when the one or more real-time Operations events are provided, performing actions including;

retrieving the one or more trained models from the memory; and

employing the one or more trained models to identify the one or more resolution metrics that are associated with the one or more real-time Operations events.

25. The network computer of claim 24, further comprising, transforming, by the one or more processors, each Operations event in the plurality of Operations events into a common event format.

26. The network computer of claim 24, further comprising, reducing, by the one or more processors, a number of Operations events included in the plurality of Operations events based on one or more of, source origin identifier value, source component identifier value, or service identifier value.

27. The network computer of claim 24, further comprising, when model performance degrades beyond accuracy requirements as measured by error produced in the output predictions, re-training, by the one or more processors, the trained model based on one or more other Operations events, one or more other event clusters, and one or more other resolution metrics.

28. The network computer of claim 24, wherein associating, by the one or more processors, the one or more resolution metrics with the one or more event clusters, further comprises, associating one or more of, a time-to-resolve value, a number of responders required to resolve, one or more names of responders or teams required to resolve, or a priority of the incident in terms of operations and business impact.

29. The network computer of claim 24, wherein providing the plurality of Operations events, further comprises, providing one or more portions of the plurality of Operations events, wherein the one or more portions of Operations events are provided by two or more separate but related sources or services.

30. The network computer of claim 24, wherein training the one or more models further comprises, employing modeling techniques, including one or more of curve-fitting, regression, statistical modeling, or machine learning to identify one or more features of the one or more event clusters that are incorporated into the one or more trained models.