System and method for automated issue remediation for information technology infrastructure

• US 10,162,698 B2
• Filed: 06/01/2016
• Issued: 12/25/2018
• Est. Priority Date: 03/25/2016
• Status: Active Grant

• Current Assignee: Dropbox, Inc.
• Original Assignee: Dropbox, Inc.
• Inventors: Mah, David, MacFiggen, Scott, Watson, John
• Primary Examiner(s): Srivastava, Vivek
• Assistant Examiner(s): McBeth, William C
• Application Number:
  US15/170,820
• Publication Number:
  US 20170279702A1
• Time in Patent Office: 937 Days
• CPC Class Codes: G06F 11/0703 Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
• CPC Class Codes: G06F 11/0748 in a remote unit communicating with a single-box computer node experiencing an error/fault remote testing G06F11/2294
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437
• CPC Class Codes: H04L 41/08 Configuration management of networks or network elements address allocation H04L61/50
• CPC Class Codes: H04L 41/0883 Semiautomatic configuration, e.g. proposals from system
• CPC Class Codes: H04L 41/0886 Fully automatic configuration
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 41/5041 characterised by the time relationship between creation and deployment of a service
• CPC Class Codes: H04L 43/12 Network monitoring probes
• CPC Class Codes: H04L 43/55 Testing of service level quality, e.g. simulating service usage
• CPC Class Codes: H04W 4/12 Messaging; Mailboxes; Announcements

Abstract

A system and method for extensible, protective, and verifiable automated issue remediation for information technology infrastructure comprises invoking an application programming interface to obtain at least one issue object corresponding to an alert generated by a monitoring system; matching the issue object to at least one diagnosis plugin of a plurality of diagnosis plugins; obtaining a prescription object from the diagnosis plugin, the prescription object comprising a remedy; and invoking the remedy after verifying the remedy is authorized to proceed.

Claims

1. A method for extensible, protective, and verifiable automated issue remediation for information technology infrastructure comprising:

invoking an application programming interface to obtain at least one issue object corresponding to an alert for a host generated by a monitoring system;

matching the issue object to at least one diagnosis plugin of a plurality of diagnosis plugins, the at least one diagnosis plugin encapsulating a first set of computer-executable instructions for execution at a host for which the alert is generated by the monitoring system, the first set of computer-executable instructions configured to diagnose the alert at the host;

based on causing the first set of computer-executable instructions encapsulated by the at least one diagnosis plugin to be executed at the host, obtaining a prescription object from the at least one diagnosis plugin, the prescription object referencing a remedy plugin, the remedy plugin encapsulating a second set of computer-executable instructions for execution at the host for which the alert is generated by the monitoring system, the second set of computer-executable instructions configured to remedy the alert at the host;

using an infrastructure protector plugin to determine whether execution of the second set of computer-executable instructions at the host should be postponed; and

based on using the infrastructure protector plugin to determine that the second set of computer-executable instructions are ready to be executed at the host, causing the second set of computer-executable instructions to be executed at the host, after verifying the second set of computer-executable instructions are authorized to be executed at the host.

2. The method of claim 1, further comprising executing the second set of computer-executable instructions at the host for which the alert is generated using a secure shell (SSH).

3. The method of claim 1, wherein the issue object comprises a set of one or more key-value pairs;

and wherein at least one of the key-value pairs in the set of one or more key-value pairs specifies an identifier of a host for which the alert is generated by the monitoring system.

4. The method of claim 1, wherein matching the issue object to the at least one diagnosis plugin comprises passing the issue object as a parameter in a call to a function of the diagnosis plugin and receiving, in response to the call, a return value indicating whether the issue object matches the diagnosis plugin.

5. The method of claim 1, wherein a prescription object comprises a reference to the at least one issue object, a reference to the diagnosis plugin, and a reference to a remedy plugin.

6. The method of claim 1, further comprising:

matching the at least one issue object to a rate-limiting rule; and

executing the second set of computer-executable instructions after postponing execution of the second set of computer-executable instructions so as not to exceed a rate threshold of the rate-limiting rule.

7. The method of claim 1, further comprising:

executing the second set of computer-executable instructions at the host after verifying that data stored at the host is replicated to a quorum of machines.

8. The method of claim 1, wherein the verifying the second set of computer-executable instructions are authorized to be executed at the host comprises checking a value of the prescription object indicating whether the second set of computer-executable instructions are automatically authorized to be executed at the host.

9. The method of claim 1, wherein the verifying the second set of computer-executable instructions are authorized to be executed at the host comprises:

presenting a description of the second set of computer-executable instructions in a user interface; and

receiving user input via the user interface specifying that the second set of computer-executable instructions are authorized to be executed at the host.

10. The method of claim 1, wherein obtaining a prescription object from the diagnosis plugin comprises calling a function of the diagnosis plugin passing the at least one issue object as a parameter and obtaining a reference to the prescription object as a return value from the function.

11. A system for extensible, protective, and verifiable automated issue remediation for information technology infrastructure comprising:

one or more processors; and

one or more storage media storing one or more computer programs for execution by the one or more processors, the one or more computer programs comprising instructions for;

invoking an application programming interface to obtain at least one issue object corresponding to an alert for a host generated by a monitoring system;

matching the issue object to at least one diagnosis plugin of a plurality of diagnosis plugins, the at least one diagnosis plugin encapsulating a first set of computer-executable instructions for execution at a host for which the alert is generated by the monitoring system, the first set of computer-executable instructions configured to diagnose the alert at the host;

based on causing the first set of computer-executable instructions encapsulated by the at least one diagnosis plugin to be executed at the host, obtaining a prescription object from the at least one diagnosis plugin, the prescription object referencing a remedy plugin, the remedy plugin encapsulating a second set of computer-executable instructions for execution at the host for which the alert is generated by the monitoring system, the second set of computer-executable instructions configured to remedy the alert at the host;

using an infrastructure protector plugin to determine whether execution of the second set of computer-executable instructions at the host should be postponed; and

based on using the infrastructure protector plugin to determine that the second set of computer-executable instructions are ready to be executed at the host, causing the second set of computer-executable instructions to be executed at the host, after verifying the second set of computer-executable instructions are authorized to be executed at the host.

12. The system of claim 11, the one or more computer programs further comprising instructions for executing the second set of computer-executable instructions at the host for which the alert is generated using a secure shell (SSH).

13. The system of claim 11, wherein the issue object comprises a set of one or more key-value pairs;

and wherein at least one of the key-value pairs in the set of one or more key-value pairs specifies an identifier of a host for which the alert is generated by the monitoring system.

14. The system of claim 11, wherein the instructions for matching the issue object to the at least one diagnosis plugin comprise instructions for:

passing the issue object as a parameter in a call to a function of the diagnosis plugin; and

receiving, in response to the call, a return value indicating whether the issue object matches the diagnosis plugin.

15. The system of claim 11, wherein a prescription object comprises a reference to the at least one issue object, a reference to the diagnosis plugin, and a reference to a remedy plugin.

16. The system of claim 11, the one or more computer programs further comprising instructions for:

matching the at least one issue object to a rate-limiting rule; and

executing the second set of computer-executable instructions after postponing execution of the second set of computer-executable instructions so as not to exceed a rate threshold of the rate-limiting rule.

17. The system of claim 11, the one or more computer programs further comprising instructions for:

executing the second set of computer-executable instructions at the host after verifying that data stored at the host is replicated to a quorum of machines.

18. The system of claim 11, wherein the instructions for verifying the second set of computer-executable instructions are authorized to be executed at the host comprise instructions for checking a value of the prescription object indicating whether the second set of computer-executable instructions are automatically authorized to be executed at the host.

19. The system of claim 11, wherein the instructions for verifying the second set of computer-executable instructions are authorized to be executed at the host comprise instructions for:

presenting a description of the second set of computer-executable instructions in a user interface; and

receiving user input via the user interface specifying that the second set of computer-executable instructions are authorized to be executed at the host.

20. The system of claim 11, wherein the instructions for obtaining a prescription object from the diagnosis plugin comprise instructions for:

calling a function of the diagnosis plugin passing the at least one issue object as a parameter; and

obtaining a reference to the prescription object as a return value from the function.