Methods and systems to tag tokens in log messages

• US 10,740,211 B2
• Filed: 11/28/2017
• Issued: 08/11/2020
• Est. Priority Date: 11/28/2017
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Brown, Darren, Kushmerick, Nicholas, Lin, Junyuan
• Primary Examiner(s): Katsikis, Kostas J
• Application Number:
  US15/824,781
• Publication Number:
  US 20190163603A1
• Time in Patent Office: 987 Days
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3075 the data filtering being achieved in order to maintain consistency among the monitored data, e.g. ensuring that the monitored data belong to the same timeframe, to the same system or component
• CPC Class Codes: G06F 11/3082 the data filtering being achieved by aggregating or compressing the monitored data
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/0686 Additional information in the notification, e.g. enhancement of specific meta-data
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 43/0823 Errors, e.g. transmission errors

Abstract

This disclosure is directed to tagging tokens or sequences of tokens in log messages generated by a logging source. Event types of log messages in a block of log messages are collected. A series of tagging operations are applied to each log message in the block. For each tagging operation, event types that are qualified to receive the corresponding tag are identified. When a log message is received, the event type is determined and compared with the event types of the block in order to identify a matching event type. The series of tagging operations are applied to the log message to generate a tagged log message with the restriction that each tagging operation only applies a tag to token or sequences of tokens when the event type is qualified to receive the tag. The tagged log message is stored in a data-storage device.

Claims

1. An automated method stored in one or more data-storage devices and executed using one or more processors of a computer system for tagging tokens and sequences of tokens in log messages generated by an event source, the method comprising:

determining tags each event type of a block of untagged log messages is qualified to receive, the block of untagged log messages generated by the event source;

performing event-type analysis on a log message generated by the event source to identify an event type of the log message;

determining an event type of the block of untagged log messages that matches the event type of the log message;

applying a series of tagging operations to tokens or sequences of tokens of the log message to generate a tagged log message, each tagging operation applying a tag to a token or a sequence of tokens of the log message when the event type of the log message matches an event type of the block of untagged log messages that is qualified to receive the tag; and

storing the tagged log message in a data-storage device.

2. The method of claim 1 wherein determining the tags each event type of untagged log messages in the block of log messages are qualified to receive comprises:

applying event-type analysis to each log message of the block of untagged log messages;

for each event type, determine log messages with the same event type in the block of untagged log messages;

applying the series of tagging operations to each log message in the block of untagged log messages; and

for each tag and for each event type,

determining a count of the log messages with the same event type and with tokens or sequences of tokens that received the same tag;

computing a fraction of log messages with the same event type and the same tag as a ratio of the count of log messages with the same event type that received the tag to the number of log messages with the same event type, and

identifying the event type as qualified to receive the tag when the fraction of log messages with the tag is greater than a tagging threshold.

3. The method of claim 1 wherein determining an event type of the block of untagged log messages that matches the event type of the log message comprises:

comparing the event type of the log message to each of the event types of the block of untagged log message; and

identifying the matching event type of the block of untagged log message as having the same non-parametric tokens as the event type of the log message.

4. The method of claim 1 wherein applying the series of tagging operations to the log message to generate the tagged log message comprises for each tagging operation in the series of tagging operations, applying the tag to tokens or sequences of tokens of the log message, when the event type of the log message matches an event type of the block of untagged log messages that is qualified to receive the tag of the tagging operation.

5. A system for tagging tokens and sequences of tokens in log messages generated by an event source, the system comprising:

one or more processors;

one or more data-storage devices; and

machine-readable instructions stored in the one or more data-storage devices that when executed using the one or more processors controls the system to perform the operations comprising;

determining tags each event type of a block of untagged log messages is qualified to receive, the block of untagged log messages generated by the event source;

performing event-type analysis on a log message generated by the event source to identify an event type of the log message;

determining an event type of the untagged block of log messages that matches the event type of the log message;

applying a series of tagging operations to tokens or sequences of tokens of the log message to generate a tagged log message, each tagging operation applying a tag to a token or a sequence of tokens of the log message when the event type of the log message matches an event type of the block of untagged log messages that is qualified to receive the tag; and

storing the tagged log message in a data-storage device.

6. The system of claim 5 wherein determining the tags each event type of untagged log messages in the block of log messages is qualified to receive comprises:

applying event-type analysis to each log message of the block of untagged log messages;

for each event type, determine log messages with the same event type in the block of untagged log messages;

applying the series of tagging operations to each log message in the block of untagged log messages; and

for each tag and for each event type,

determining a count of the log messages with the same event type and with tokens or sequences of tokens that received the same tag;

computing a fraction of log messages with the same event type and the same tag as a ratio of the count of log messages with the same event type that received the tag to the number of log messages with the same event type, and

identifying the event type as qualified to receive the tag when the fraction of log messages with the tag is greater than a tagging threshold.

7. The system of claim 5 wherein determining an event type of the block of untagged log messages that matches the event type of the log message comprises:

comparing the event type of the log message to each of the event types of the block of untagged log message; and

identifying the matching event type of the block of untagged log message as having the same non-parametric tokens as the event type of the log message.

8. The system of claim 5 wherein applying the series of tagging operations to the log message to generate the tagged log message comprises for each tagging operation in the series of tagging operations, applying the tag to tokens or sequences of tokens of the log message, when the event type of the log message matches an event type of the block of untagged log messages that is qualified to receive the tag of the tagging operation.

9. A non-transitory computer-readable medium encoded with machine-readable instructions that implement a method carried out by one or more processors of a computer system to perform the operations comprising:

determining tags each event type of a block of untagged log messages is qualified to receive, the block of untagged log messages generated by the event source;

performing event-type analysis on a log message generated by the event source to identify an event type of the log message;

determining an event type of the block of untagged log messages that matches the event type of the log message;

applying a series of tagging operations to tokens or sequences of tokens of the log message to generate a tagged log message, each tagging operation applying a tag to a token or a sequence of tokens of the log message when the event type of the log message matches an event type of the block of untagged log messages that is qualified to receive the tag; and

storing the tagged log message in a data-storage device.

10. The medium of claim 9 wherein determining the tags each event type of log messages in the block of log messages are qualified to receive comprises:

applying event-type analysis to each log message of the block of untagged log messages;

for each event type, determine log messages with the same event type in the block of untagged log messages;

applying the series of tagging operations to each log message in the block of untagged log messages; and

for each tag and for each event type,

determining a count of the log messages with the same event type and with tokens or sequences of tokens that received the same tag;

computing a fraction of log messages with the same event type and the same tag as a ratio of the count of log messages with the same event type that received the tag to the number of log messages with the same event type, and

identifying the event type as qualified to receive the tag when the fraction of log messages with the tag is greater than a tagging threshold.

11. The medium of claim 9 wherein determining an event type of the block of untagged log messages that matches the event type of the log message comprises:

comparing the event type of the log message to each of the event types of the block of untagged log message; and

identifying the matching event type of the block of untagged log message as having the same non-parametric tokens as the event type of the log message.

12. The medium of claim 9 wherein applying the series of tagging operations to the log message to generate the tagged log message comprises for each tagging operation in the series of tagging operations, applying the tag to tokens or sequences of tokens of the log message, when the event type of the log message matches an event type of the block of untagged log messages that is qualified to receive the tag of the tagging operation.