Generation and use of trained file classifiers for malware detection

• US 9,864,956 B1
• Filed: 05/01/2017
• Issued: 01/09/2018
• Est. Priority Date: 05/01/2017
• Status: Active Grant

• Current Assignee: SparkCognition Inc.
• Original Assignee: SparkCognition Inc.
• Inventors: Sai, Na
• Primary Examiner(s): Armouche, Hadi
• Assistant Examiner(s): Zarrineh, Shahriar
• Application Number:
  US15/583,565
• Time in Patent Office: 253 Days
• Field of Search: 726 23
• CPC Class Codes: G06F 21/562 Static detection
• CPC Class Codes: G06F 2221/033 Test or assess software
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms

Abstract

A method includes training a file classifier from one or more n-gram feature vectors received from a plurality of binary files as input, where the one or more n-gram vectors represent the occurrences of character pairs in printable characters within the file or characters representing the informational entropy sequence of the file. Another method also includes generating, by the file classifier, output including classification data associated with the file based on the one or more n-gram vectors, where the classification data indicates whether the file includes malware.

Claims

1. A computing device comprising:

a memory configured to store instructions to execute a trained file classifier; and

a processor configured to execute the instructions from the memory to perform operations comprising;

receiving, via a network from a remote computing device, a feature vector representing a file stored in a memory of the remote computing device, the feature vector including;

a zero-skip n-gram indicating occurrences of adjacent characters in printable characters representing the file,

a skip n-gram indicating occurrences of non-adjacent characters in the printable characters representing the file; and

an n-gram indicating occurrences of groups of entropy indicators in a set of entropy indicators derived from file entropy data for the file, each entropy indicator of the set of entropy indicators having a value representing entropy of a corresponding chunk of the file;

generating, by the trained file classifier, classification data associated with the file based on the feature vector, the classification data indicating whether the file includes malware; and

transmitting the classification data to the remote computing device via the network, wherein access to the file or execution of the file at the remote computing device is restricted responsive to the classification data indicating that the file includes malware.

2. The computing device of claim 1, wherein the zero-skip n-gram includes a Boolean vector indicating the occurrences of the adjacent characters in the printable characters.

3. The computing device of claim 1, wherein the zero-skip n-gram indicates a count of the occurrences of the adjacent characters in the printable characters.

4. The computing device of claim 1, wherein the skip n-gram includes a Boolean vector indicating the occurrences of the non-adjacent characters in the printable characters.

5. The computing device of claim 1, wherein the trained file classifier corresponds to a decision tree, a neural network, or a support vector machine.

6. The computing device of claim 1, wherein the skip n-gram indicates a count of the occurrences of the non-adjacent characters in the printable characters.

7. A method comprising:

receiving, via a network from a remote computing device, a feature vector representing a file stored in a memory of the remote computing device, the feature vector including;

a zero-skip n-gram indicating occurrences of adjacent characters in printable characters representing the file,

a skip n-gram indicating occurrences of non-adjacent characters in the printable characters representing the file; and

an n-gram indicating occurrences of groups of entropy indicators in a set of entropy indicators derived from file entropy data for the file, each entropy indicator of the set of entropy indicators having a value representing entropy of a corresponding chunk of the file;

generating, by a trained file classifier, classification data associated with the file based on the feature vector, the classification data indicating whether the file includes malware; and

transmitting the classification data to the remote computing device via the network, wherein access to the file or execution of the file at the remote computing device is restricted responsive to the classification data indicating that the file includes malware.

8. The method of claim 7, further comprising processing the file to generate the printable characters representing the file.

9. The method of claim 7, wherein the zero-skip n-gram includes a Boolean vector indicating the occurrences of the adjacent characters in the printable characters.

10. The method of claim 7, wherein the zero-skip n-gram indicates a count of the occurrences of the adjacent characters in the printable characters.

11. The method of claim 7, wherein the feature vector includes a plurality of skip n-grams including the skip n-gram, each skip n-gram of the plurality of skip n-grams based on a different skip value.

12. The method of claim 7, wherein the non-adjacent characters of the skip n-gram correspond to two characters separated by at least one other character in the printable characters.

13. The method of claim 7, wherein the non-adjacent characters of the skip n-gram correspond to two characters separated by at least two other characters in the printable characters.

14. The method of claim 7, wherein the non-adjacent characters of the skip n-gram to two characters separated by at least three other characters in the printable characters.

15. The method of claim 7, further comprising comparing a hash value determined from the feature vector to file identifiers stored in a memory, wherein the classification data is generated by the trained file classifier based on a determination that the hash value does not match one of the file identifiers.

16. A computer-readable storage device storing instructions that, when executed, cause a computer to perform operations comprising:

receiving, via a network from a remote computing device, a feature vector representing a file stored in a memory of the remote computing device, the feature vector including;

a zero-skip n-gram indicating occurrences of adjacent characters in printable characters representing the file,

a skip n-gram indicating occurrences of non-adjacent characters in the printable characters representing the file; and

an n-gram indicating occurrences of groups of entropy indicators in a set of entropy indicators derived from file entropy data for the file, each entropy indicator of the set of entropy indicators having a value representing entropy of a corresponding chunk of the file;

generating, by a trained file classifier, output including classification data associated with the file based on the feature vector, the classification data indicating whether the file includes malware; and

transmitting the classification data to the remote computing device via the network, wherein access to the file or execution of the file at the remote computing device is restricted responsive to the classification data indicating that the file includes malware.

17. The computer-readable storage device of claim 16, wherein the n-gram indicating occurrences of groups of entropy indicators is based on value of n that is different from a value of n of the zero-skip n-gram and is different from a value of n of the skip n-gram.

18. The computer-readable storage device of claim 16, wherein the operations further comprise comparing a hash value determined from the feature vector to file identifiers in a memory, wherein the classification data is generated by the trained file classifier based on a determination that the hash value does not match one of the file identifiers.

19. The computer-readable storage device of claim 18, wherein the operations further comprise generating the hash value based on the feature vector before comparing the hash value to the file identifiers in the memory.

20. The computer-readable storage device of claim 16, wherein the file is a binary file and the printable characters representing the binary file are generated by converting at least a portion of the binary file to American Standard Code for Information Interchange (ASCII) characters.