Systems and methods for filtering low utility value messages from system logs

• US 9,535,981 B2
• Filed: 07/15/2013
• Issued: 01/03/2017
• Est. Priority Date: 07/15/2013
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Basak, Jayanta, Chandrasekarasastry, Nagesh Panyam
• Primary Examiner(s): Leroux, Etienne
• Assistant Examiner(s): Nguyen, Cindy
• Application Number:
  US13/942,392
• Publication Number:
  US 20150019512A1
• Time in Patent Office: 1,268 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/335 Filtering based on additional data, e.g. user or group profiles filtering in web context G06F16/9535, G06F16/9536

Abstract

Systems and methods disclosed herein provide intelligent filtering of system log messages having low utility value. In providing the filtering, the systems and methods determine the utility value of a system log message and delete the message from the system log if the message is determined to be of low utility value. As such, embodiments herein provide an system log filter, which reduces the amount of data stored in the system log based on the utility value of the message.

Claims

1. A computer-implemented method comprising:

determining, by a log filter, a utility value of each message of a plurality of messages of a system log, the utility value of a respective message based on predictability of information in the respective message, the predictability of information based on a set of one or more messages preceding the respective message in the systemlog;

purging each respective message of the plurality of messages which are determined to have a utility values that is lower than a threshold utility value;

wherein before purging messages determined to have a low utility value;

identifying duplicative messages within one or more respective subsequence of the plurality of messages;

deleting, within each respective subsequence of the plurality of messages, all but one of the duplicative messages;

after deleting all but one of the duplicative messages within each respective subsequence of the plurality of messages;

computing a utility score for each remaining message of the system log,

comparing the utility score of each remaining message to the threshold utility value, wherein each message having a utility score lower than the threshold utility value is determined to have a low utility value,

identifying subsequences of the remaining messages, wherein the subsequences of the remaining messages comprise a threshold number of messages of low utility value, and

determining, for each of said subsequences of the remaining messages, that no duplicative messages exist within the respective subsequences of the remaining messages;

and

storing, in a memory, each respective message of the plurality of messages which are determined to have a utility value that is equal to or higher than the threshold utility value.

2. The method of claim 1 wherein the log filter determines that the utility value of the respective message is lower than the threshold utility value at least when the log filter is able to predict information of the respective message from the set of one or more messages preceding the respective message in the system log.

3. The method of claim 1 wherein the set of one or more messages preceding the respective message in the system log is a determined number of messages that immediately precede the respective message in the system log.

4. The method of claim 3 wherein the determined number of messages is dynamically determined based at least on an availability of one or more system resources.

5. The method of claim 1 further comprising:

determining the system log'"'"'s buffer size.

6. The method of claim 5 wherein the threshold utility value is dynamically set based at least on the determined size of the buffer of the system log.

7. The method of claim 1 wherein the threshold utility value is dynamically set based at least on an availability of one or more system resources.

8. The method of claim 1 further comprising:

sending the stored messages to a remote system.

9. A system comprising:

a processor;

a log filter for execution by the processor, the log filter operable to;

determine a utility value of each message of a plurality of messages logged in a system log, the utility value of a respective message based on predictability of information in the respective message, the predictability of information based on a set of one or more messages preceding the respective message in the system log;

purge each respective message of the plurality of messages which are determined to have a utility value that is lower than a threshold utility value,

wherein before messages determined to have a low utility value are purged;

identify duplicative messages within one or more respective subsequence of the plurality of messages;

delete, within each respective subsequence of the plurality of messages, all but one of the duplicative messages;

after deleting all but one of the duplicative messages within each respective subsequence of the plurality of messages;

compute a utility score for each remaining message of the system log,

compare the utility score of each remaining message to the threshold utility value, wherein each message having a utility score lower than the threshold utility value is determined to have a low utility value,

identify subsequences of the remaining messages, wherein the subsequences of the remaining messages comprise a threshold number of messages of low utility value, and

determine, for each of said subsequences of the remaining messages, that no duplicative messages exist within the respective subsequences of the remaining messages;

and

store each respective message of the plurality of messages which are determined to have a utility value that is equal to or higher than the threshold utility value.

10. The system of claim 9 wherein the log filter is further operable to determine that the utility value of a respective message is lower than the threshold utility value at least when the log filter is able to predict information of the respective message from the set of one or more messages preceding the respective message in the system log.

11. The system of claim 9 wherein the set of one or more messages preceding the respective message in the system log is a determined number of messages that immediately precede the respective message in the system log.

12. The system of claim 11 wherein the determined number of messages is dynamically determined based at least on an availability of one or more system resources.

13. The system of claim 9 wherein the log filter is further operable to:

determine the system log'"'"'s buffer size.

14. The system of claim 13 wherein the threshold utility value is dynamically set based at least on the determined buffer size of the system log.

15. The system of claim 9 wherein the threshold utility value is dynamically set based at least on an availability of one or more system resources.

16. The system of claim 9 further comprising:

a transmitter operable to send the stored messages to a remote system.

17. A computer-implemented method comprising:

computing, by a log filter, a utility score for each message of a plurality of messages of a system log, the utility score of a respective message based on predictability of information in the respective message, the predictability of information based on a set of one or more messages preceding the respective message in the system log;

comparing the utility score of each message to a threshold utility value, wherein each message having a utility score lower than the threshold utility value is determined to have a low utility value; and

deleting, from a memory, all messages determined to have a low utility value,

wherein before deleting all messages determined to have a low utility value;

identifying duplicative messages within one or more respective subsequence of the plurality of messages;

deleting, within each respective subsequence of the plurality of messages, all but one of the duplicative messages;

after deleting all but one of the duplicative messages within each respective subsequence of the plurality of messages;

computing a utility score for each remaining message of the system log,

comparing the utility score of each remaining message to the threshold utility value, wherein each message having a utility score lower than the threshold utility value is determined to have a low utility value,

identifying subsequences of the remaining messages, wherein the subsequences of the remaining messages comprise a threshold number of messages of low utility value, and determining, for each of said subsequences of the remaining messages, that no duplicative messages exist within the respective subsequences of the remaining messages.

18. The method of claim 17 further comprising:

before deleting all messages determined to have a low utility value;

identifying subsequences of the plurality of messages, wherein the subsequences comprise a threshold number of messages of low utility value;

determining, for each said subsequence of the plurality of messages, that no duplicative messages exist within the respective subsequence of the plurality of messages.

19. The method of claim 17 wherein the threshold utility score is dynamically determined based at least on an availability of one or more system resources.