Methods and apparatus to improve feature engineering efficiency with metadata unit operations

• US 10,607,004 B2
• Filed: 09/29/2016
• Issued: 03/31/2020
• Est. Priority Date: 09/29/2016
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Yang, Chih-Yuan, Gai, Yi
• Primary Examiner(s): Gee, Jason K
• Assistant Examiner(s): Torres-Diaz, Lizbeth
• Application Number:
  US15/280,044
• Publication Number:
  US 20180089424A1
• Time in Patent Office: 1,279 Days
• CPC Class Codes: G06F 21/52 during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]

Abstract

Methods, apparatus, systems and articles of manufacture are disclosed to improve feature engineering efficiency. An example method disclosed herein includes retrieving a log file in a first file format, the log file containing feature occurrence data, generating a first unit operation based on the first file format to extract the feature occurrence data from the log file to a string, the first unit operation associated with a first metadata tag, generating second unit operations to identify respective features from the feature occurrence data, the second unit operations associated with respective second metadata tags, and generating a first sequence of the first metadata tag and the second metadata tags to create a first vector output file of the feature occurrence data.

Claims

1. A computer-implemented method to apply pattern engineering with metadata-driven unit operations to improve an efficiency of analysis of log files having different file formats, the computer-implemented method comprising:

determining, by executing an instruction with a processor, a first file format of a log file from the different file formats, the log file to be converted to a vector output file;

generating, by executing an instruction with the processor, a first sequence of processing tasks based on the first file format, the first sequence including a first metadata tag corresponding to a conversion task to convert the first file format to a string format, the log file including pattern occurrence data;

generating, by executing an instruction with the processor, a second metadata tag within the first sequence of processing tasks, the second metadata tag associated with a second processing task to identify respective features from the pattern occurrence data by comparing the string to a pattern corresponding to malware;

generating, by executing an instruction with the processor, a third metadata tag within the first sequence of processing tasks, the third metadata tag associated with a third processing task to create the vector output file of the respective features from the pattern occurrence data identified by the second metadata tag; and

executing, by executing an instruction with the processor, the first sequence of the first metadata tag, the second metadata tag, and the third metadata tag to create the vector output file associated with the first file format.

2. The computer-implemented method as defined in claim 1, wherein the first metadata tag includes parsing operations of the log file for at least one of a text file format, a comma separated value (CSV) file format, a JavaScript Object Notation (JSON) file format, or a binary file format.

3. The computer-implemented method as defined in claim 1, further including building a dictionary of feature nomenclature associated with the respective features from the pattern occurrence data.

4. The computer-implemented method as defined in claim 3, further including generating search substrings of the feature nomenclature.

5. The computer-implemented method as defined in claim 4, wherein the second metadata tag identifies at least one of the search substrings as feature occurrence instances.

6. The computer-implemented method as defined in claim 1, further including executing a second sequence of the first metadata tag, a fourth metadata tag, and a fifth metadata tag to create a second vector output file associated with the first file format.

7. The computer-implemented method as defined in claim 6, wherein the second metadata tag is to invoke a dictionary to identify feature occurrence instances, and the fourth metadata tag is to invoke regular expressions to identify feature occurrence instances.

8. An apparatus to apply pattern engineering with metadata-driven metadata tags to improve an efficiency of analysis of log files having different file formats, the apparatus comprising:

a log file retriever to;

retrieve a log file in a first log file format, the log file including pattern occurrence data; and

determine a first file format of the log file from the different file formats, the log file to be converted to a vector output file;

a file to string operation builder to generate a first metadata tag of a first sequence of processing tasks that is based on the first file format, the first metadata tag corresponding to a conversion task to convert the first file format to a string format;

an extraction operation builder to generate a second metadata tag within the first sequence of processing tasks, the second metadata tag associated with a second processing task to identify respective features from the pattern occurrence data by comparing the string to a pattern corresponding to malware;

a feature save operation builder to generate a third metadata tag within the first sequence of processing tasks, the third metadata tag associated with a third processing task to save the vector output file; and

an operation flow builder to;

generate the first sequence of processing tasks, the first sequence including the first metadata tag, the second metadata tag, and the third metadata tag to create the vector output file of the respective features from the pattern occurrence data identified by the second metadata tag; and

execute the first sequence of the first metadata tag, the second metadata tag, and the third metadata tag to create the vector output file associated with the first file format.

9. The apparatus as defined in claim 8, wherein the file to string operation builder is to generate parsing operations of the log file for at least one of a text file format, a comma separated value (CSV) file format, a JavaScript Object Notation (JSON) file format, or a binary file format.

10. The apparatus as defined in claim 8, further including a dictionary editor to build a dictionary of feature nomenclature associated with the respective features from the pattern occurrence data.

11. The apparatus as defined in claim 8, further including a dictionary editor to facilitate extraction of feature instances based on at least one of dictionary matching or regular expression strings.

12. The apparatus as defined in claim 8, further including a dictionary editor to normalize one or more search substrings to identify feature nomenclature based on a dictionary association.

13. The apparatus as defined in claim 8, further including a hashing operation builder to hash the respective features to a unique integer value.

14. The apparatus as defined in claim 8, further including a formatting operation builder to format the vector output file based on a Library for Support Vector Machines (LIBSVM) classification format.

15. A tangible computer readable storage medium comprising computer readable instructions to improve an efficiency of analysis of log files having different file formats, the instructions which, when executed, cause a processor to at least:

determine a first file format of a log file from the different file formats, the log file to be converted to a vector output file;

generate a first sequence of processing tasks based on the first file format, the first sequence including a first metadata tag corresponding to a conversion task to convert the first file format to a string format, the log file including pattern occurrence data;

generate a second metadata tag within the first sequence of processing tasks, the second metadata tag associated with a second processing task to identify respective features from the pattern occurrence data by comparing the string to a pattern corresponding to malware;

generate a third metadata tag within the first sequence of processing tasks, the third metadata tag associated with a third processing task to create the vector output file of the respective features from the pattern occurrence data identified by the second metadata tag; and

execute the first sequence of the first metadata tag, the second metadata tag, and the third metadata tag to create the vector output file associated with the first file format.

16. The tangible computer readable storage medium of claim 15, wherein the instructions, when executed, cause the processor to generate parsing operations of the log file for at least one of a text file format, a comma separated value (CSV) file format, a JavaScript Object Notation (JSON) file format, or a binary file format.

17. The tangible computer readable storage medium of claim 15, wherein the instructions, when executed, cause the processor to build a dictionary of feature nomenclature associated with the respective features from the pattern occurrence data.

18. The tangible computer readable storage medium of claim 17, wherein the instructions, when executed, cause the processor to generate search substrings of the feature nomenclature.

19. The tangible computer readable storage medium of claim 18, wherein the instructions, when executed, cause the processor to identify, by executing the second processing task associated with the second metadata tag, the search substrings as feature occurrence instances.

20. The tangible computer readable storage medium of claim 15, wherein the instructions, when executed, cause the processor to execute a second sequence of the first metadata tag, a fourth metadata tag, and a fifth metadata tag to create a second vector output file associated with the first file format.