LOGGING INCIDENT MANAGER

• US 20150207709A1
• Filed: 01/21/2014
• Published: 07/23/2015
• Est. Priority Date: 01/21/2014
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: HANCKEL, Robert, THILAGAR, Parthiban
• Granted Patent: US 9,742,624 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04L 41/0609 based on severity or priority
• CPC Class Codes: H04L 41/0618 based on the physical or logical position
• CPC Class Codes: H04L 41/0622 based on time
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications

Abstract

Systems, methods, and other embodiments associated with intelligently processing log messages are described. In one embodiment, a computer-implemented method includes analyzing, by a logging appliance that includes at least hardware, communications received from a plurality of handlers to determine whether at least one of the communications indicates an error has been encountered by one of a plurality of components associated with the plurality of handlers. The method includes scheduling a subset of the plurality of handlers to provide reports that include a detailed set of log messages in response to detecting the error. The subset of the plurality of handlers includes handlers that are associated with the error.

Claims

1. A computer-implemented method associated with processing log messages from a plurality of handlers that are distributed among a plurality of hosts in an application cluster, the method comprising:

receiving, at a logging appliance that includes at least a processor, communications from the plurality of handlers, wherein the communications include log messages for a plurality of components associated with the plurality of handlers, and wherein the application cluster includes the plurality of components distributed among the plurality of hosts;

analyzing the communications to determine whether at least one of the communications indicates an error has been encountered by one of the plurality of components; and

scheduling a subset of the plurality of handlers to provide reports in response to detecting the error, wherein the subset of the plurality of handlers includes handlers that are associated with the error.

2. The computer-implemented method of claim 1, wherein the error is a fault condition in one of the plurality of components that has been registered in a log message by one of the plurality of handlers, wherein the reports include a detailed set of log messages, wherein scheduling the subset of the plurality handlers to provide the reports includes selecting handlers from the plurality of handlers that are likely to have information that is relevant to the error,

wherein selecting the handlers is based, at least in part, on a relationship of one or more of the plurality of handlers to a component of the plurality of components that generated the error, and

wherein selecting the handlers is based, at least in part, on whether a handler reported a secondary error proximate to the error and whether the handler has log messages with time stamps proximate to a timestamp of the error.

3. The computer-implemented method of claim 1, wherein the detailed set of log messages included with each of the reports is a batch of log messages gathered according to a fine granularity of logging at a time that is proximate to the error,

wherein the communications include batches of log messages gathered according to a coarse level of granularity, wherein the reports include log messages gathered at a finer granularity than the communications, and

wherein scheduling the subset of the plurality of handlers to provide the reports includes specifying a time for each handler of the subset to provide a report to the logging appliance by providing a request to each handler of the subset.

4. The computer-implemented method of claim 1, wherein each of the communications includes a batch of log messages gathered according to a coarse level of logging, and wherein the communications are heartbeat communications from the plurality of handlers that are communicated at a regular interval.

5. The computer-implemented method of claim 1, further comprising:

prior to scheduling the subset of the plurality of handlers to provide reports, waiting a predefined period of time to ensure communications from the plurality of handlers with log messages that have timestamps from prior to the error have been received, wherein the predefined period of time is an integer multiple of a regular interval at which the plurality of handlers provide the communications.

6. The computer-implemented method of claim 1, further comprising:

generating a summary of the error from at least the reports received from the subset of the plurality of handlers, wherein the summary is generated after expiration of a waiting period for receiving the reports, wherein the waiting period accounts for time skew between the plurality of handlers and delays in receiving the reports, wherein the summary is generated from one or more of the reports and communications received proximate to a time of the error, and wherein the reports include log messages collected at a fine granularity and the communications include log messages collected at a coarse granularity.

7. The computer-implemented method of claim 1, further comprising:

clearing the error from a state of the application cluster and returning the application cluster to a clear operating state after a predetermined period of time elapses without additional errors occurring.

8. The computer-implemented method of claim 1, further comprising:

storing the communications and the reports in a distributed memory associated with the logging appliance, wherein the communications and the reports include log messages from the plurality of handlers, and wherein storing the communications and the reports in the distributed memory includes sorting and collating the log messages according to timestamps.

9. The computer-implemented method of claim 1, wherein scheduling the subset of the plurality of handlers to provide reports includes requesting the subset to freeze incident buffers that include log messages collected at a fine granularity and requesting the subset to communicate the log messages from the incident buffers at a scheduled time, wherein receiving the communications includes receiving at least one communication prior to an expected time and in response to an associated handler logging an error.

10. A computing system for processing log messages from a plurality of handlers that are distributed among a plurality of hosts in an application cluster, comprising:

a message logic configured to receive communications from the plurality of handlers, wherein the communications include log messages about a plurality of components associated with the plurality of handlers, and wherein the application cluster includes the plurality of components distributed among the plurality of hosts; and

incident manager logic configured to analyze the communications to determine whether at least one of the communications indicates an error has been encountered by one of the plurality of components, wherein the error is a fault condition in one of the plurality of components that has been registered in a log message by one of the plurality of handlers, and

wherein the incident management logic is configured to schedule a subset of the plurality of handlers to provide reports that include a detailed set of log messages in response to detecting the error, wherein the subset of the plurality of handlers includes handlers that are associated with the error.

11. The computing system of claim 10, wherein the incident manager logic is configured to schedule the subset of the plurality handlers to provide the reports by selecting handlers from the plurality of handlers that are likely to have information that is relevant to the error,

wherein the incident manager logic is configured to select the handlers based, at least in part, on a relationship of one or more of the plurality of handlers to a component of the plurality of components that generated the error, and

wherein the incident manager logic is configured to select the handlers according to whether a handler reported a secondary error proximate to the error and whether the handler has log messages with time stamps proximate to a timestamp of the error.

12. The computing system of claim 10, wherein the detailed set of log messages included with each of the reports is a batch of log messages gathered according to a fine granularity of logging at a time that is proximate to the error, wherein the communications include batches of log messages gathered according to a coarse level of granularity, wherein the reports include log messages gathered at a finer granularity than the communications, and

wherein the incident manager logic is configured to schedule the subset of the plurality of handlers to provide the reports according to a time specified for each handler of the subset to communicate a report to the message logic as specified in a control communication provided by the incident manager logic.

13. The computing system of claim 10, wherein each of the communications includes a batch of log messages gathered according to a coarse level of granularity, and wherein the communications are heartbeat communications from the plurality of handlers that are communicated at a regular interval.

14. The computing system of claim 10, wherein the incident manager logic is configured to, prior to scheduling the subset of the plurality of handlers to provide reports, wait a predefined period of time to ensure communications from the plurality of handlers that have timestamps from prior to the error have been received, and wherein the predefined period of time is an integer multiple of a regular interval at which the plurality of handlers provide the communications.

15. The computing system of claim 10, wherein the incident manager logic is configured to generate a summary of the error from at least the reports received from the subset of the plurality of handlers,

wherein the incident manager logic is configured to generate the summary after expiration of a waiting period for receiving the reports, wherein the waiting period accounts for time skew between the plurality of handlers and delays in receiving the reports,

wherein the summary is generated from the reports and communications received proximate to a time of the error, and wherein the reports include log messages collected at a fine granularity and the communications include log messages collected at a coarse granularity.

16. The computing system of claim 10, wherein the incident manager logic is configured to clear the error from a state of the application cluster and return the application cluster to a clear operating state after a predetermined period of time elapses without additional errors occurring.

17. The computing system of claim 10, wherein the incident manager logic is configured to store the communications and the reports in a distributed memory, wherein the communications and the reports include log messages from the plurality of handlers, and wherein the incident manager logic is configured to sort and collate the log messages according to timestamps after storing the log messages in the distributed memory.

18. The computing system of claim 10, wherein the incident manager is configured to schedule the subset of the plurality of handlers to provide reports by requesting the subset to freeze incident buffers that include log messages collected at a fine granularity and requesting the subset to communicate the log messages from the incident buffers at a scheduled time, and

wherein the incident manager logic is configured to receive the communications by receiving at least one communication prior to an expected time and in response to an associated handler of the plurality of handlers logging an error.

19. A non-transitory computer-storage medium storing logic for execution by a device for processing log messages from a component that is part of an application cluster, the non-transitory computer-storage medium comprising:

logic configured to analyze communications received from a plurality of handlers to determine whether at least one of the communications indicates an error has been encountered by one of a plurality of components associated with the plurality of handlers; and

logic configured to schedule a subset of the plurality of handlers to provide reports that include a detailed set of log messages in response to detecting the error, wherein the subset of the plurality of handlers includes handlers that are associated with the error.

20. The computer-implemented method of claim 19, wherein the logic configured to schedule the subset of the plurality handlers to provide the reports includes logic configured to select handlers from the plurality of handlers that are likely to have information that is relevant to the error,

wherein the detailed set of log messages included with each of the reports is a batch of log messages gathered according to a fine granularity of logging at a time that is proximate to the error, and

wherein the communications include batches of log messages gathered according to a coarse level of granularity, wherein the reports include log messages gathered at a finer granularity than the communications.