Management of log data

• US 9,934,265 B2
• Filed: 04/09/2015
• Issued: 04/03/2018
• Est. Priority Date: 04/09/2015
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Karandikar, Amey Vijaykumar, Vibhor, Anand, Upadhyay, Mrityunjay
• Primary Examiner(s): Le, Debbie
• Application Number:
  US14/683,005
• Publication Number:
  US 20160299934A1
• Time in Patent Office: 1,090 Days
• Field of Search: None
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/248 Presentation of query results

Abstract

A system according to certain aspects improves the process of managing log data. The system may receive a log data file including one or more log lines that contain information relating to computing operations. At least some of the log lines may include a static portion and a variable portion. The system may process a first log line to identify and extract a static portion from the first log line, and determine a first value for the first log line based on the extracted static portion. The system may process a second log line to identify and extract a static portion from the second log line, and determine a second value for the second log line based on the extracted static portion. The system may compare the first and second values, and based on the comparison, organize the first and second log lines together for presentation to a user.

Claims

1. A method for processing log data, the method comprising:

with one or more computing devices comprising computer hardware,

receiving a log data file comprising one or more log lines, the one or more log lines including information relating to one or more computing operations, wherein at least some of the one or more log lines include a static portion and a variable portion;

processing a first log line of the one or more log lines to identify a static portion of the first log line;

extracting the static portion from the first log line;

extracting a first set of one or more trigrams from the extracted static portion of the first log line;

calculating a value for each trigram in the first set of one or more trigrams;

determining a first value by aggregating the values calculated for the first set of one or more trigrams, wherein the values calculated for the first set of one or more trigrams are aggregated by calculating a sum of the values;

processing a second log line of the one or more log lines to identify a static portion of the second log line;

extracting the static portion from the second log line;

extracting a second set of one or more trigrams from the extracted static portion of the second log line;

calculating a value for each trigram in the second set of one or more trigrams;

determining a second value by aggregating the values calculated for the second set of one or more trigrams, wherein the values calculated for the second set of one or more trigrams are aggregated by calculating a sum of the values;

comparing the first and second values; and

based on the comparison, organizing the first and second log lines together for presentation to a user.

2. The method of claim 1, wherein the first and second values are the same and uniquely identify the static portion of text of the first log line and the second log line.

3. The method of claim 1, wherein said organizing comprises adding the first and second log lines to a group of log lines based on the first and second values, wherein the log lines in the group of log lines are associated with a unique set of values that are not associated with any other group.

4. The method of claim 1, further comprising:

extracting a variable portion from the first log line of the one or more log lines;

determining a third value for the first log line based on the extracted variable portion;

extracting a variable portion from the second log line of the one or more log lines; and

determining a fourth value for the second log line based on the extracted variable portion.

5. The method of claim 4, wherein said organizing comprises:

adding the first and second log lines to a group of log lines based on the first and second values, wherein the log lines in the group of log lines are associated with values that are identical to the first and second values; and

adding the first and second log lines to a sub-group within the group of log lines based on the third and fourth values, wherein the log lines in the sub-group are associated with values that are identical to the third and fourth values.

6. The method of claim 1, wherein said determining of the first and second values comprises determining the first and second values using a hash function configured to generate a value based on the extracted static portions of the first and second log lines, respectively.

7. The method of claim 1, wherein the static portions of the first and second log lines indicate the types of transaction associated with the first and second log lines, respectively, and variable portions of the first and second log lines indicate additional information regarding the transactions associated with the first and second log lines, respectively.

8. The method of claim 1, wherein said extracting of the static portion comprises determining a sequence of characters that have been processed more than a threshold number of times in the history of processing the log data.

9. A system for processing log data, the system comprising:

a computing device comprising computer hardware and configured to;

receive a log data file comprising one or more log lines, the one or more log lines including information relating to one or more computing operations, wherein at least some of the one or more log lines include a static portion and a variable portion;

process a first log line of the one or more log lines to identify a static portion of the first log line;

extract the static portion from the first log line;

extract a first set of one or more trigrams from the extracted static portion of the first log line;

calculate a value for each trigram in the first set of one or more trigrams;

determine a first value by aggregating the values calculated for the first set of one or more trigrams, wherein the values calculated for the first set of one or more trigrams are aggregated by calculating a sum of the values;

process a second log line of the one or more log lines to identify a static portion of the second log line;

extract the static portion from the second log line;

extract a second set of one or more trigrams from the extracted static portion of the second log line;

calculate a value for each trigram in the second set of one or more trigrams;

determine a second value by aggregating the values calculated for the second set of one or more trigrams, wherein the values calculated for the second set of one or more trigrams are aggregated by calculating a sum of the values;

compare the first and second values; and

based on the comparison, organize the first and second log lines together for presentation to a user.

10. The system of claim 9, wherein the first and second values are the same and uniquely identify the static portion of text of the first log line and the second log line.

11. The system of claim 9, wherein the computing device is configured to organize the first and second log lines at least by adding the first and second log lines to a group of log lines based on the first and second values, wherein the log lines in the group of log lines are associated with a unique set of values that are not associated with any other group.

12. The system of claim 9, wherein the computing device is further configured to:

extract a variable portion from the first log line of the one or more log lines;

determine a third value for the first log line based on the extracted variable portion;

extract a variable portion from the second log line of the one or more log lines; and

determine a fourth value for the second log line based on the extracted variable portion.

13. The system of claim 12, wherein the computing device is configured to organize the first and second log lines at least by:

adding the first and second log lines to a group of log lines based on the first and second values, wherein the log lines in the group of log lines are associated with values that are identical to the first and second values; and

adding the first and second log lines to a sub-group within the group of log lines based on the third and fourth values, wherein the log lines in the sub-group are associated with values that are identical to the third and fourth values.

14. The system of claim 9, wherein the computing device is configured to determine the first and second values at least by determining the first and second values using a hash function configured to generate a value based on the extracted static portions of the first and second log lines, respectively.

15. The system of claim 9, wherein the static portions of the first and second log lines indicate the types of transaction associated with the first and second log lines, respectively, and variable portions of the first and second log lines indicate additional information regarding the transactions associated with the first and second log lines, respectively.

16. The system of claim 9, wherein the computing device is configured to extract the static portion at least by determining a sequence of characters that have been processed more than a threshold number of times in the history of processing the log data.