Generation and use of trained file classifiers for malware detection

• US 10,068,187 B1
• Filed: 05/31/2017
• Issued: 09/04/2018
• Est. Priority Date: 05/01/2017
• Status: Active Grant

• Current Assignee: SparkCognition Inc.
• Original Assignee: SparkCognition Inc.
• Inventors: Sai, Na
• Primary Examiner(s): Shiferaw, Eleni A
• Assistant Examiner(s): Zarrineh, Shahriar
• Application Number:
  US15/610,173
• Time in Patent Office: 461 Days
• Field of Search: 726 23
• CPC Class Codes: G06F 21/562 Static detection
• CPC Class Codes: G06F 2221/033 Test or assess software
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms

Abstract

A method includes accessing information identifying multiple files and identifying classification data for the multiple files, where the classification data indicates, for a particular file of the multiple files, whether the particular file includes malware. The method also includes generating n-gram vectors for the multiple files by, for each file, generating an n-gram vector indicating occurrences of character pairs in printable characters representing the file. The method further includes generating and storing a file classifier using the n-gram vectors and the classification data as supervised training data.

Claims

1. A computing device comprising:

a memory configured to store instructions to generate a trained file classifier; and

a processor configured to execute the instructions from the memory to perform operations comprising;

accessing information identifying multiple files and identifying classification data for the multiple files, wherein the classification data indicates, for a particular file of the multiple files, whether the particular file includes malware;

generating a feature vector representing the particular file of the multiple files, the feature vector including;

zero-skip n-gram data indicating occurrences of adjacent characters in printable characters representing the particular file;

skip n-gram data indicating occurrences of non-adjacent characters in the printable characters representing the particular file; and

n-gram data indicating occurrences of groups of entropy indicators in a set of entropy indicators derived from file entropy data for the particular file, each entropy indicator of the set of entropy indicators having a value representing entropy of a corresponding chunk of the particular file;

generating the trained file classifier using the feature vector and the classification data as supervised training data; and

transmitting the trained file classifier to a remote computing device via a network, wherein the trained file classifier is executable by the remote computing device to restrict access to a file or to restrict execution of the file based on a classification result generated by execution of the trained file classifier.

2. The computing device of claim 1, wherein the zero-skip n-gram data includes a Boolean vector indicating the occurrences of the adjacent characters in the printable characters.

3. The computing device of claim 1, wherein the zero-skip n-gram data indicates a count of the occurrences of the adjacent characters pairs in the printable characters.

4. The computing device of claim 1, wherein the skip n-gram data includes a Boolean vector indicating the occurrences of the non-adjacent characters in the printable characters.

5. The computing device of claim 1, wherein the trained file classifier corresponds to a decision tree, a neural network, or a support vector machine.

6. The computing device of claim 1, wherein the skip n-gram data indicates a count of the occurrences of the non-adjacent characters in the printable characters.

7. A method comprising:

accessing information identifying multiple files and identifying classification data for the multiple files, wherein the classification data indicates, for a particular file of the multiple files, whether the particular file includes malware;

generating a feature vector representing the particular file of the multiple files, the feature vector including;

zero-skip n-gram data indicating occurrences of adjacent characters in printable characters representing the particular file;

skip n-gram data indicating occurrences of non-adjacent characters in the printable characters representing the particular file; and

n-gram data indicating occurrences of groups of entropy indicators in a set of entropy indicators derived from file entropy data for the particular file, each entropy indicator of the set of entropy indicators having a value representing entropy of a corresponding chunk of the particular file;

generating a trained file classifier using the feature vector and the classification data as supervised training data; and

transmitting the trained file classifier to a remote computing device via a network, wherein the trained file classifier is executable by the remote computing device to restrict access to a file or to restrict execution of the file based on a classification result generated by execution of the trained file classifier.

8. The method of claim 7, further comprising processing the particular file to generate the printable characters representing the particular file.

9. The method of claim 7, wherein the zero-skip n-gram data includes a Boolean vector indicating the occurrences of the adjacent characters in the printable characters.

10. The method of claim 7, wherein the zero-skip n-gram data indicates a count of the occurrences of the adjacent characters in the printable characters.

11. The method of claim 7, wherein the feature vector includes data representing to a plurality of skip n-grams, each skip n-gram of the plurality of skip n-grams based on a different skip value.

12. The method of claim 7, wherein the non-adjacent characters of the skip n-gram data correspond to two characters separated by at least one other character in the printable characters.

13. The method of claim 7, wherein the non-adjacent characters of the skip n-gram data correspond to two characters separated by at least two other characters in the printable characters.

14. The method of claim 7, wherein the non-adjacent characters of the skip n-gram data correspond to two characters separated by at least three other characters in the printable characters.

15. The method of claim 7, wherein the feature vector includes data representing to a plurality of skip n-grams, each skip n-gram of the plurality of skip n-grams based on a different skip value, and wherein the n-gram data indicating occurrences of the groups of entropy indicators is based on a value of n that is different from a value of n of zero-skip n-grams corresponding to the zero-skip n-gram data.

16. A computer-readable storage device storing instructions that, when executed, cause a computer to perform operations comprising:

accessing information identifying multiple files and identifying classification data for the multiple files, wherein the classification data indicates, for a particular file of the multiple files, whether the particular file includes malware;

generating a feature vector representing the particular file of the multiple files, the feature vector including;

zero-skip n-gram data indicating occurrences of adjacent characters in printable characters representing the particular file;

skip n-gram data indicating occurrences of non-adjacent characters in the printable characters representing the particular file; and

n-gram data indicating occurrences of groups of entropy indicators in a set of entropy indicators derived from file entropy data for the particular file, each entropy indicator of the set of entropy indicators having a value representing entropy of a corresponding chunk of the particular file;

generating and storing a trained file classifier using the feature vector and the classification data as supervised training data; and

causing the trained file classifier to be transmitted to a remote computing device via a network, wherein the trained file classifier is executable by the remote computing device to restrict access to a file or to restrict execution of the file based on a classification result generated by execution of the trained file classifier.

17. The computer-readable storage device of claim 16, wherein the multiple files include files with malware, files without malware, and corresponding classification data.

18. The computer-readable storage device of claim 16, wherein the operations further comprise:

generating a hash value based on the feature vector and storing the hash value of the feature vector as an identifier of the particular file.

19. The computer-readable storage device of claim 16, wherein the particular file is a binary file and the operations further comprise generating the printable characters representing the binary file by converting at least a portion of the binary file to American Standard Code for Information Interchange (ASCII) characters.

20. The computer-readable storage device of claim 16, wherein the feature vector includes data representing a plurality of skip n-grams, each skip n-gram of the plurality of skip n-grams based on a different skip value.