Generating important values from a variety of server log files

• US 10,394,868 B2
• Filed: 10/23/2015
• Issued: 08/27/2019
• Est. Priority Date: 10/23/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: He, Miao, Li, Jin Feng, Ren, Chang Rui, Shao, Bing, Xie, Ming, Zhao, Tian Zhi
• Primary Examiner(s): Harper, Eliyah S.
• Application Number:
  US14/921,288
• Publication Number:
  US 20170116330A1
• Time in Patent Office: 1,404 Days
• Field of Search: 707648
• CPC Class Codes: G06F 16/3322 using system suggestions G06F16/3325 takes precedence
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/35 Clustering; Classification

Abstract

A mechanism is provided for identifying suggestions as to which log files associated with data in a data warehouse to search for particular data. Features from the contents of a plurality of log files from a set of log files are selected. The plurality of log files are grouped based on the selected features. Using extracted log event sequences, a correlation between the log event sequences and a plurality of data transaction tables from a set of data transaction tables in the data warehouse is calculated. Suggestions as to which log files in the set of log files should be searched is then identified for particular data based on a business relevance value and a utilized data ratio.

Claims

1. A method, in a data processing system comprising a process and a memory coupled to the processor, for identifying suggestions as to which log files associated with data in a data warehouse to search for particular data, the method comprising:

utilizing an identified data structure of a plurality of log files from a set of log files, selecting, by feature selection logic specifically configured by a log file evaluation mechanism within the data processing system, features from the contents of the log file;

grouping, by the feature selection logic, log files in the set of log files together based on the selected features;

from structured data of the grouped log files, extracting, by extraction logic specifically configured by the log file evaluation mechanism within the data processing system, log event sequences;

calculating, by correlation logic specifically configured by the log file evaluation mechanism within the data processing system, a correlation between the log event sequences and a plurality of data transaction tables from a set of data transaction tables in the data warehouse;

utilizing a highest valued correlate log sequence for the plurality of data transaction tables, determining, by the correlation logic, a business relevance value between the plurality of log files and a business analysis objective, wherein determining the business relevance value between each log file in the set of log files and the business analysis objective utilizes the following business relevance function;

2. The method of claim 1, wherein calculating the correlation between the log event sequences and the plurality of data transaction tables comprises:

reading, by the correlation logic, metadata associated with the plurality of data transaction tables from a data dictionary;

identifying, by the correlation logic, an associated data transaction table in the data warehouse;

reading, by the correlation logic, data sequences from the data transaction table;

calculating, by the correlation logic, a relevance score between each log event sequence and the data transaction tables in the data warehouse thereby forming a set of relevance scores; and

determining, by the correlation logic, a maximum relevance score from the set of relevance scores.

3. The method of claim 2, wherein calculating the relevance score between each log event sequence and the plurality of data transaction tables uses the following correlation score function:

4. The method of claim 3, wherein determining the maximum relevance score from the set of relevance scores uses the following correlation value function:

relevance score r(log file,table)=max(p(log sequence,table)).

5. The method of claim 1, wherein determining the utilized data ratio that indicates the proportion of the data in the data warehouse that has been interacted with as per the associated log file utilizes the following business relevance function:

6. The method of claim 1, wherein the identified data structure of the plurality of log files in a set of log files is identified by parsing, by parsing logic specifically configured by the log file evaluation mechanism within the data processing system, each log file to identify the structured data.

7. The method of claim 1, wherein grouping the log files in the set of log files together based on the selected features is performed using an Euclidean distance analysis, a Manhattan distance analysis, or text similarity.

8. A non-transitory computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

utilizing an identified data structure of a plurality of log file from a set of log files, select, utilizing feature selection logic of a log file evaluation mechanism specifically configured by readable program, features from the contents of the log file;

group, utilizing the feature selection logic, log files in the set of log files together based on the selected features;

from structured data of the grouped log files, extract, utilizing extraction logic of the log file evaluation mechanism specifically configured by the computer readable program, log event sequences;

calculate, utilizing correlation logic of the log file evaluation mechanism specifically configured by the computer readable program, a correlation between the log event sequences and a plurality of data transaction tables from a set of data transaction tables in the data warehouse;

utilizing a highest valued correlate log sequence for the plurality of data transaction tables, determine, utilizing the correlation logic, a business relevance value between the plurality of log files and a business analysis objective, wherein the computer readable program to determine the business relevance value between each log file in the set of log files and the business analysis objective utilizes the following business relevance function;

9. The non-transitory computer readable storage medium of claim 8, wherein the computer readable program to calculate the correlation between the log event sequences and the plurality of data transaction tables further causes the computing device to:

read, utilizing the correlation logic, metadata associated with the plurality of data transaction tables from a data dictionary;

identify, utilizing the correlation logic, an associated data transaction table in the data warehouse;

read, utilizing the correlation logic, data sequences from the data transaction table;

calculate, utilizing the correlation logic, a relevance score between each log event sequence and the data transaction tables in the data warehouse thereby forming a set of relevance scores; and

determine, utilizing the correlation logic, a maximum relevance score from the set of relevance scores.

10. The non-transitory computer readable storage medium of claim 9, wherein the computer readable program to calculate the relevance score between each log event sequence and the plurality of data transaction tables uses the following correlation score function:

11. The non-transitory computer readable storage medium of claim 10, wherein the computer readable program to determine the maximum relevance score from the set of relevance scores uses the following correlation value function:

relevance score r(log file,table)=max(p(log sequence,table)).

12. The non-transitory computer readable storage medium of claim 8, wherein the computer readable program to determine the utilized data ratio that indicates the proportion of the data in the data warehouse that has been interacted with as per the associated log file utilizes the following business relevance function:

13. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

utilizing an identified data structure of a plurality of log files from a set of log files, select, utilizing feature selection logic of a log file evaluation mechanism specifically configured by the instructions, features from the contents of the log file;

group, utilizing the feature selection logic, log files in the set of log files together based on the selected features;

from structured data of the grouped log files, extract, utilizing extraction logic of the log file evaluation mechanism specifically configured by the instructions, log event sequences;

calculate, utilizing correlation logic of the log file evaluation mechanism specifically configured by instructions, a correlation between the log event sequences and a plurality of data transaction tables from a set of data transaction tables in the data warehouse;

utilizing a highest valued correlate log sequence for the plurality of data transaction tables, determine, utilizing the correlation logic, a business relevance value between the plurality of log files and a business analysis objective, wherein the instructions to determine the business relevance value between each log file in the set of log files and the business analysis objective utilizes the following business relevance function;

14. The apparatus of claim 13, wherein the instructions to calculate the correlation between the log event sequences and the plurality of data transaction tables further cause the processor to:

read, utilizing the correlation logic, metadata associated with the plurality of data transaction tables from a data dictionary;

identify, utilizing the correlation logic, an associated data transaction table in the data warehouse;

read, utilizing the correlation logic, data sequences from the data transaction table;

calculate, utilizing the correlation logic, a relevance score between each log event sequence and the data transaction tables in the data warehouse thereby forming a set of relevance scores; and

determine, utilizing the correlation logic, a maximum relevance score from the set of relevance scores.

15. The apparatus of claim 14, wherein the instructions to calculate the relevance score between each log event sequence and the plurality of data transaction tables uses the following correlation score function:

16. The apparatus of claim 15, wherein the instructions to determine the maximum relevance score from the set of relevance scores uses the following correlation value function:

relevance score r(log file,table)=max(p(log sequence,table)).

17. The apparatus of claim 13, wherein the instructions to determine the utilized data ratio that indicates the proportion of the data in the data warehouse that has been interacted with as per the associated log file utilizes the following business relevance function: