Triaging computing systems

• US 9,529,652 B2
• Filed: 11/07/2014
• Issued: 12/27/2016
• Est. Priority Date: 11/07/2013
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Doppalapudi, Raghu Chaitanya
• Primary Examiner(s): Ehne, Charles
• Application Number:
  US14/535,550
• Publication Number:
  US 20150127979A1
• Time in Patent Office: 781 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0715 in a system implementing multitasking multitasking per se G06F9/46
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/302 where the computing system component is a software system
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence

Abstract

Methods and systems are provided for automatically triaging a server cluster of the type including a plurality of linked servers each running a plurality of processes. The method includes: detecting at least one failed process; automatically transmitting an electronic alert message embodying a first error code indicative of the failed process to a unified triage module including a processor and an updatable index table; applying, by the processor, the first error code to the index table. If a matching error code corresponding to the first error code is found in the index table, retrieving a solution code from the index table associated with the matching error code and automatically restarting the failed process using the solution code without human intervention.

Claims

1. A method of operating a server cluster of the type including a plurality of linked servers each running a plurality of processes, the method comprising:

detecting at least one failed process;

automatically transmitting an electronic alert message embodying a first error code indicative of the failed process to a unified triage module including a processor and an updatable index table, wherein the alert message identifies the at least one failed process and the linked server running the at least one failed process;

applying, by the processor, the first error code to the index table;

if a matching error code corresponding to the first error code is found in the index table, retrieving a solution code from the index table associated with the matching error code; and

automatically restarting the failed process using the solution code without human intervention.

2. The method of claim 1, further comprising:

in response to detecting the failed process;

automatically retrieving a log file associated with the failed process; and

electronically transmitting the log file to the unified triage module.

3. The method of claim 2, wherein the log file comprises operational data temporally coincident with the failed process.

4. The method of claim 3, wherein temporally coincident corresponds to a predetermined time range surrounding the failure event associated with the failed process.

5. The method of claim 4, wherein temporally coincident corresponds to a predetermined time range surrounding the failure event associated with the failed process.

6. The method of claim 5, wherein the predetermined time range includes approximately one hour before and one hour after the failure event.

7. The method of claim 1, wherein the alert message further identifies the cluster to which the linked server belongs.

8. The method of claim 2, further comprising:

if a matching error code corresponding to the first error code is not found in the index table, transmitting the alert message and the log file to a user interface of the type configured to facilitate manually triaging the failed process.

9. The method of claim 8, further comprising:

updating the index table to reflect the result of the manual triaging.

10. The method of claim 8, wherein the log file comprises at least one of a graph, a report, and a dashboard visualization relating to the failed process.

11. The method of claim 1, wherein the updatable index table comprises a plurality of objects each corresponding to a previously failed process and a corresponding solution.

12. The method of claim 1, wherein detecting comprises simultaneously monitoring the plurality of processes by periodically polling each of the plurality of linked servers.

19. The processing system of claim 10, wherein the updatable index table comprises a plurality of objects each corresponding to a previously failed storage process and a corresponding solution.

13. A processing system for triaging failures in an on-demand computing environment, comprising:

a database system configured to run a plurality of storage processes and to record associated log data;

a unified triage (UT) module that includes an index table and a set of proposed solutions to one or more failed storage processes, the index table configured to identify the one or more proposed solutions;

a monitoring module configured to listen to the database system and to detect a failed storage process, the monitoring module further configured to transmit a corresponding alert to the UT module when a failed storage process is detected; and

an analytics module connected to the database system and configured to generate a log file based on the log data, wherein the log file comprises operational data temporally coincident with the failed storage process, and to transmit the log file to the UT module upon receipt by the UT module of the alert;

wherein the UT module is configured to retrieve a proposed solution when it receives the alert from the monitoring module, wherein the proposed solution is retrieved based on data stored in the log file and by using solution from the index table to access a corresponding solution.

14. The processing system of claim 13, wherein the UT module is further configured to restart the failed storage process using the proposed solution.

15. The processing system of claim 13, further comprising a user interface configured to facilitate manual triaging of the failed storage process.

16. The processing system of claim 15, wherein the UT module is further configured to electronically transmit the alert and the log file to the user interface if the index table does not contain a proposed solution.

17. The processing system of claim 16, wherein the user interface is further configured to update the index table to reflect successful manual triaging of the failed storage process.

18. The processing system of claim 16, wherein the alert transmitted to the UT module identifies the failed storage processes and one of a plurality of linked servers running the failed storage process.

20. A non-transitory computer readable medium comprising computer readable instructions that, when executed by a processor, perform the steps comprising:

detecting a failed process in a server cluster of the type including a plurality of linked servers;

automatically transmitting an electronic alert message and a log file each corresponding to the failed process to a unified triage module, wherein the electronic alert message identifies the failed process and the linked server running the failed process;

searching an index table for a solution to the failed process;

if a solution is found in the index table, automatically restarting the failed process using the solution;

if a solution is found in the index table, transmitting the alert message and the log file to a user interface for manually triaging the failed process; and

updating the index table using the unified triage module to reflect the results of the manual triaging.