Management of log data

• US 10,296,613 B2
• Filed: 03/31/2017
• Issued: 05/21/2019
• Est. Priority Date: 04/09/2015
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Karandikar, Amey Vijaykumar, Vibhor, Anand, Upadhyay, Mrityunjay
• Primary Examiner(s): Le, Debbie M
• Application Number:
  US15/475,746
• Publication Number:
  US 20170206235A1
• Time in Patent Office: 781 Days
• Field of Search: None
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/248 Presentation of query results

Abstract

A system according to certain aspects improves the process of managing log data. The system may receive a log data file including one or more log lines that contain information relating to computing operations. At least some of the log lines may include a static portion and a variable portion. The system may process a first log line to identify and extract a static portion from the first log line, and determine a first value for the first log line based on the extracted static portion. The system may process a second log line to identify and extract a static portion from the second log line, and determine a second value for the second log line based on the extracted static portion. The system may compare the first and second values, and based on the comparison, organize the first and second log lines together for presentation to a user.

Claims

1. A method for processing log data, the method comprising:

with one or more computing devices comprising computer hardware,

receiving a log data file comprising one or more log lines, the log lines including information relating to one or more computing operations;

processing a first log line of the one or more log lines to identify a first variable portion of the first log line;

extracting a first static portion of the first log line;

extracting the first variable portion from the first log line;

determining a first value for the first log line by using a hash function configured to generate a value based on the extracted first static portion;

processing a second log line of the one or more log lines to identify a second variable portion of the second log line;

extracting the second variable portion from the second log line;

extracting a second static portion of the second log line;

determining a second value for the second log line by using a hash function configured to generate a value based on the extracted second static portion;

comparing the first and second values;

based on the comparison, if the first and second values match, grouping the first and second log lines together;

comparing the first and second variable portions; and

modifying the static portions of the first and second log lines if the first variable portion matches the second variable portion.

2. The method of claim 1, wherein the first and second values are the same and uniquely identify the variable portion of text of the first log line and the second log line.

3. The method of claim 1, wherein the grouping further comprises adding the first and second log lines to a group of log lines based on the first and second values, wherein the log lines in the group of log lines are associated with a unique set of values that are not associated with any other group.

4. The method of claim 1, wherein said grouping further comprises:

adding the first and second log lines to a group of log lines based on the first and second values, wherein the log lines in the group of log lines are associated with values that are identical to the first and second values; and

adding the first and second log lines to a sub-group within the group of log lines based on third and fourth values, wherein the log lines in the sub-group are associated with values that are identical to the third and fourth values.

5. The method of claim 1, wherein said comparing of the first and second variable portions comprises computing the values using a hash function configured to generate a value based on the first and second variable portions, respectively.

6. The method of claim 1, wherein said comparing of the first and second variable portions comprises:

extracting a first set of one or more trigrams from the first variable portion and a second set of one or more trigrams from the second variable portion;

calculating a value for each of the first and second sets of one or more trigrams; and

aggregating the values calculated for the first and second sets of one or more trigrams, respectively;

comparing the values calculated for the first and second sets of one or more trigrams.

7. The method of claim 6, wherein the values calculated for the first and second sets of one or more trigrams are aggregated by calculating a sum of the values.

8. The method of claim 1, wherein the static portions of the first and second log lines indicate types of transaction associated with the first and second log lines, respectively, and variable portions of the first and second log lines indicate additional information regarding the transactions associated with the first and second log lines, respectively.

9. The method of claim 1, wherein said extracting of the variable portion comprises determining a sequence of characters that have been processed more than a threshold number of times in the history of processing the log data.

10. A system for processing log data, the system comprising:

a computing device comprising computer hardware and configured to;

receive a log data file comprising one or more log lines, the log lines including information relating to one or more computing operations, wherein at least some of the log lines include a variable portion and a static portion;

process a first log line of the one or more log lines to identify a first variable portion of the first log line;

extract the first variable portion from the first log line;

extract a first static portion from the first log line;

determine a first value for the first log line by using a hash function configured to generate a value based on the extracted first static portion;

process a second log line of the one or more log lines to identify a second variable portion of the second log line;

extract the second variable portion from the second log line;

extract a second static portion from the second log line;

determine a second value for the second log line by using a hash function configured to generate a value based on the extracted second static portion;

compare the first and second values;

based on the comparison, if the first and second values match, group the first and second log lines together for presentation to a user;

compare the first and second variable portions; and

modify the static portions of the first and second log lines if the first variable portion matches the second variable portion.

11. The system of claim 10, wherein the first and second values are the same and uniquely identify the variable portion of text of the first log line and the second log line.

12. The system of claim 10, wherein the computing device is configured to organize the first and second log lines at least by adding the first and second log lines to a group of log lines based on the first and second values, wherein the log lines in the group of log lines are associated with a unique set of values that are not associated with any other group.

13. The system of claim 10, wherein the computing device is configured to group the first and second log lines at least by:

adding the first and second log lines to a group of log lines based on the first and second values, wherein the log lines in the group of log lines are associated with values that are identical to the first and second values; and

adding the first and second log lines to a sub-group within the group of log lines based on third and fourth values, wherein the log lines in the sub-group are associated with values that are identical to the third and fourth values.

14. The system of claim 10, wherein the computing device is configured to compare the first and second variable portions at least by computing values using a hash function configured to generate a value based on the extracted first and second variable portions, respectively.

15. The system of claim 10, wherein the computing device is configured to compare the first and second variable portions at least by:

extracting a first set of one or more trigrams from the first variable portion and a second set of one or more trigrams from the second variable portion;

calculating a value for each of the first and second sets of one or more trigrams; and

aggregating the values calculated for the first and second sets of one or more trigrams, respectively; and

comparing the values calculated for the first and second sets of one or more trigrams.

16. The system of claim 15, wherein the values calculated for the first and second sets of one or more trigrams are aggregated by calculating a sum of the values.

17. The system of claim 10, wherein the static portions of the first and second log lines indicate types of transaction associated with the first and second log lines, respectively, and variable portions of the first and second log lines indicate additional information regarding the transactions associated with the first and second log lines, respectively.

18. The system of claim 10, wherein the computing device is configured to extract the variable portion at least by determining a sequence of characters that have been processed more than a threshold number of times in the history of processing the log data.