Real-time adaptive operations performance management system using event clusters and trained models

• US 9,811,795 B1
• Filed: 02/27/2017
• Issued: 11/07/2017
• Est. Priority Date: 09/01/2016
• Status: Active Grant

• Current Assignee: Pagerduty, Inc.
• Original Assignee: Pagerduty, Inc.
• Inventors: Kearns, Justin David, Ronen, Ophir, Zuchlewski, Laura Ann
• Primary Examiner(s): Vincent, David
• Application Number:
  US15/443,961
• Time in Patent Office: 253 Days
• Field of Search: 706 15, 706 45
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/0637 Strategic management or analysis, e.g. setting a goal or target of an organisation; Planning actions based on goals; Analysis or evaluation of effectiveness of goals

Abstract

Embodiments are directed to managing operations. If Operations events are provided, event clusters may be associated with one or more Operations events, such that the Operations events may be associated with the event clusters based on characteristics of the Operations events. Metrics including resolution metrics, root cause analysis, notes, and other remediation information may be associated with the event clusters. Then a modeling engine may be employed to train models based on the Operations events, the event clusters, and the resolution metrics, such that the trained model may be trained to correlate and predict the resolution metrics from real-time Operations events. If real-time Operations events may be provided, the trained models may be employed to predict the resolution metrics that are associated with the real-time Operations events. If model performance degrades beyond accuracy requirements, new observations may be added to the training set and the model re-trained.

Claims

1. A method for managing operations for organizations over a network using one or more network computers that include one or more processors that perform actions, comprising:

employing a plurality of provided Operations events to perform further actions, including;

providing, by the one or more processors, one or more event clusters that are associated with one or more Operations events;

associating, by the one or more processors, one or more resolution metrics with the one or more event clusters;

employing, by the one or more processors, a modeling engine to train one or more models based on the one or more Operations events, the one or more event clusters, and the one or more resolution metrics, wherein the one or more trained models are stored in a datastore; and

retrieving, by the one or more processors, the one or more trained models, from the datastore, that are used to identify the one or more resolution metrics that are associated with one or more real-time Operations events.

2. The method of claim 1, further comprising determining and separating the one or more event clusters by one or more values based on one or more of time, frequency or spatial coordinates.

3. The method of claim 1, further comprising providing a user interface for a human user to perform one or more actions, including providing the association of the one or more resolution metrics with the one or more event clusters, or quantifying one or more features of the one or more event clusters.

4. The method of claim 1, further comprising employing each event cluster that is correlated with the one or more real time Operation events to provide one or more of event cluster information, associated Operation events, or resolution metrics for use as a training set to evaluate the one or more trained models.

5. The method of claim 1, further comprising training the one or more training models based on characterizing and measuring one or more of coefficients, convergence values and error values for the one or more training models.

6. The method of claim 1, further comprising grouping the one or more real time Operation events in the one or more event clusters based on time and frequency of the one or more Operation events.

7. The method of claim 1, further comprising normalizing the plurality of Operations events with a data structure that provides a common format.

8. The method of claim 1, wherein the trained model is configured to correlate and predict the one or more resolution metrics from the one or more real-time Operations events.

9. A system for managing operations for organizations over a network, comprising:

a network computer, comprising;

a transceiver that communicates over the network;

a memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including;

employing a plurality of provided Operations events to perform further actions, including;

providing, by the one or more processors, one or more event clusters that are associated with one or more Operations events;

 

associating, by the one or more processors, one or more resolution metrics with the one or more event clusters;

employing, by the one or more processors, a modeling engine to train one or more models based on the one or more Operations events, the one or more event clusters, and the one or more resolution metrics, wherein the one or more trained models are stored in a datastore; and

retrieving, by the one or more processors, the one or more trained models, from the datastore, that are used to identify the one or more resolution metrics that are associated with one or more real-time Operations events; and

a client computer, comprising;

a client computer transceiver that communicates over the network;

a client computer memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including;

displaying the plurality of Operations events;

obtaining user inputs for associating the one or more resolution metrics with the one or more event clusters; and

communicating the user inputs to the network computer.

10. The system of claim 9, further comprising determining and separating the one or more event clusters by one or more values based on one or more of time, frequency or spatial coordinates.

11. The system of claim 9, further comprising providing a user interface for a human user to perform one or more actions, including providing the association of the one or more resolution metrics with the one or more event clusters, or quantifying one or more features of the one or more event clusters.

12. The system of claim 9, further comprising employing each event cluster that is correlated with the one or more real time Operation events to provide one or more of event cluster information, associated Operation events, or resolution metrics for use as a training set to evaluate the one or more trained models.

13. The system of claim 9, further comprising training the one or more training models based on characterizing and measuring one or more of coefficients, convergence values and error values for the one or more training models.

14. The system of claim 9, further comprising grouping the one or more real time Operation events in the one or more event clusters based on time and frequency of the one or more Operation events.

15. The system of claim 9, further comprising normalizing the plurality of Operations events with a data structure that provides a common format.

16. The system of claim 9, wherein the trained model is configured to correlate and predict the one or more resolution metrics from the one or more real-time Operations events.

17. A processor readable non-transitory storage media that includes instructions for managing operations for organizations over a network, wherein execution of the instructions by one or more processors performs actions, comprising:

employing a plurality of provided Operations events to perform further actions, including;

providing, by the one or more processors, one or more event clusters that are associated with one or more Operations events;

associating, by the one or more processors, one or more resolution metrics with the one or more event clusters;

employing, by the one or more processors, a modeling engine to train one or more models based on the one or more Operations events, the one or more event clusters, and the one or more resolution metrics, wherein the one or more trained models are stored in a datastore; and

retrieving, by the one or more processors, the one or more trained models, from the datastore, that are used to identify the one or more resolution metrics that are associated with one or more real-time Operations events.

18. The media of claim 17, further comprising determining and separating the one or more event clusters by one or more values based on one or more of time, frequency or spatial coordinates.

19. The media of claim 17, further comprising providing a user interface for a human user to perform one or more actions, including providing the association of the one or more resolution metrics with the one or more event clusters, or quantifying one or more features of the one or more event clusters.

20. The media of claim 17, further comprising employing each event cluster that is correlated with the one or more real time Operation events to provide one or more of event cluster information, associated Operation events, or resolution metrics for use as a training set to evaluate the one or more trained models.

21. The media of claim 17, further comprising training the one or more training models based on characterizing and measuring one or more of coefficients, convergence values and error values for the one or more training models.

22. The media of claim 17, further comprising grouping the one or more real time Operation events in the one or more event clusters based on time and frequency of the one or more Operation events.

23. The media of claim 17, further comprising normalizing the plurality of Operations events with a data structure that provides a common format.

24. The media of claim 17, wherein the trained model is configured to correlate and predict the one or more resolution metrics from the one or more real-time Operations events.

25. A network computer for managing operations for organizations over a network, comprising:

a transceiver that communicates over the network;

a memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including;

employing a plurality of provided Operations events to perform further actions, including;

providing, by the one or more processors, one or more event clusters that are associated with one or more Operations events;

associating, by the one or more processors, one or more resolution metrics with the one or more event clusters;

employing, by the one or more processors, a modeling engine to train one or more models based on the one or more Operations events, the one or more event clusters, and the one or more resolution metrics, wherein the one or more trained models are stored in a datastore; and

retrieving, by the one or more processors, the one or more trained models, from the datastore, that are used to identify the one or more resolution metrics that are associated with one or more real-time Operations events.

26. The network computer of claim 25, further comprising determining and separating the one or more event clusters by one or more values based on one or more of time, frequency or spatial coordinates.

27. The network computer of claim 25, further comprising providing a user interface for a human user to perform one or more actions, including providing the association of the one or more resolution metrics with the one or more event clusters, or quantifying one or more features of the one or more event clusters.

28. The network computer of claim 25, further comprising employing each event cluster that is correlated with the one or more real time Operation events to provide one or more of event cluster information, associated Operation events, or resolution metrics for use as a training set to evaluate the one or more trained models.

29. The network computer of claim 25, further comprising training the one or more training models based on characterizing and measuring one or more of coefficients, convergence values and error values for the one or more training models.

30. The network computer of claim 25, further comprising grouping the one or more real time Operation events in the one or more event clusters based on time and frequency of the one or more Operation events.