Generation and use of trained file classifiers for malware detection

• US 10,062,038 B1
• Filed: 05/31/2017
• Issued: 08/28/2018
• Est. Priority Date: 05/01/2017
• Status: Active Grant

• Current Assignee: SparkCognition Inc.
• Original Assignee: SparkCognition Inc.
• Inventors: Sai, Na
• Primary Examiner(s): Shiferaw, Eleni A
• Assistant Examiner(s): Zarrineh, Shahriar
• Application Number:
  US15/610,228
• Time in Patent Office: 454 Days
• CPC Class Codes: G06F 21/562 Static detection
• CPC Class Codes: G06F 2221/033 Test or assess software
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms

Abstract

A method includes accessing information identifying multiple files and identifying classification data for the multiple files, where the classification data indicates, for a particular file of the multiple files, whether the particular file includes malware. The method also includes generating a sequence of entropy indicators for each of the multiple files, each entropy indicator of the sequence of entropy indicators for the particular file corresponding to a chunk of the particular file. The method further includes generating n-gram vectors for the multiple files, where the n-gram vector for the particular file indicates occurrences of groups of entropy indicators in the sequence of entropy indicators for the particular file. The method also includes generating and storing a file classifier using the n-gram vectors and the classification data as supervised training data.

Claims

1. A computing device comprising:

a memory configured to store instructions to generate a file classifier; and

a processor configured to execute the instructions from the memory to perform operations comprising;

accessing information identifying multiple files and identifying classification data for the multiple files, wherein the classification data indicates, for a particular file of the multiple files, whether the particular file includes malware;

generating a sequence of entropy indicators for each of the multiple files, each entropy indicator of the sequence of entropy indicators for the particular file corresponding to a chunk of the particular file;

generating n-gram vectors for the multiple files, wherein an n-gram vector for the particular file indicates occurrences of groups of entropy indicators in the sequence of entropy indicators for the particular file; and

generating and storing a file classifier using the n-gram vectors and the classification data as supervised training data, wherein the supervised training data includes a plurality of n-gram vectors for each file, the plurality of n-gram vectors for at least one file including a zero-skip n-gram vector indicating occurrences of groups of adjacent entropy indicators, and including at least one skip n-gram vector indicating occurrences of groups of non-adjacent entropy indicators.

2. The computing device of claim 1, wherein the n-gram vectors include a Boolean vector indicating the occurrences of the groups of entropy indicators.

3. The computing device of claim 1, wherein the n-gram vectors include a count of the occurrences of the groups of entropy indicators.

4. The computing device of claim 1, wherein each group of entropy indicators of the groups of entropy indicators includes two or more entropy indicators.

5. The computing device of claim 1, wherein the file classifier corresponds to a decision tree, a neural network, or a support vector machine.

6. The computing device of claim 1, wherein the operations further comprise generating printable characters corresponding to each of the multiple files, and wherein the file classifier is generated further based on the printable characters.

7. A method comprising:

accessing information identifying multiple files and identifying classification data for the multiple files, wherein the classification data indicates, for a particular file of the multiple files, whether the particular file includes malware;

generating a sequence of entropy indicators for each of the multiple files, each entropy indicator of the sequence of entropy indicators for the particular file corresponding to a chunk of the particular file;

generating n-gram vectors for the multiple files, where an n-gram vector for the particular file indicates occurrences of groups of entropy indicators in the sequence of entropy indicators for the particular file; and

generating and storing a file classifier using the n-gram vectors and the classification data as supervised training data, wherein the supervised training data includes a plurality of n-gram vectors for each file, the plurality of n-gram vectors for at least one file including a zero-skip n-gram vector indicating occurrences of groups of adjacent entropy indicators, and including at least one skip n-gram vector indicating occurrences of groups of non-adjacent entropy indicators.

8. The method of claim 7, wherein the multiple files include files with malware, files without malware, and corresponding classification data.

9. The method of claim 7, wherein the n-gram vectors include a Boolean vector indicating the occurrences of the groups of entropy indicators.

10. The method of claim 7, wherein the n-gram vector includes a count of the occurrences of the groups of entropy indicators.

11. The method of claim 7, wherein a group of entropy indicators of the groups of entropy indicators corresponds to two adjacent entropy indicators.

12. The method of claim 7, further comprising generating printable characters corresponding to each of the multiple files, and wherein the file classifier is generated further based on the printable characters.

13. The method of claim 7, wherein generating the n-gram vectors includes generating, for each file, a plurality of n-gram vectors, the plurality of n-gram vectors for each file including a zero-skip n-gram vector indicating the occurrences of the groups of adjacent entropy indicators and including multiple n-gram vectors indicating occurrences of pairs of printable characters.

14. A computer-readable storage device storing instructions that, when executed, cause a computer to perform operations comprising:

accessing information identifying multiple files and identifying classification data for the multiple files, wherein the classification data indicates, for a particular file of the multiple files, whether the particular file includes malware;

generating a sequence of entropy indicators for each of the multiple files, each entropy indicator of the sequence of entropy indicators for the particular file corresponding to a chunk of the particular file;

generating n-gram vectors for the multiple files, wherein an n-gram vector for the particular file indicates occurrences of groups of entropy indicators in the sequence of entropy indicators for the particular file; and

generating and storing a file classifier using the n-gram vectors and the classification data as supervised training data, wherein the supervised training data includes a plurality of n-gram vectors for each file, the plurality of n-gram vectors for at least one file including a zero-skip n-gram vector indicating occurrences of groups of adjacent entropy indicators, and including at least one skip n-gram vector indicating occurrences of groups of non-adjacent entropy indicators.

15. The computer-readable storage device of claim 14, wherein the multiple files include files with malware, files without malware, and corresponding classification data.

16. The computer-readable storage device of claim 14, wherein the operations further comprise generating a plurality of printable characters representing the file, and wherein generating the n-gram vectors includes generating, for at least one file, an n-gram vector indicating occurrences of pairs of printable characters.

17. The computer-readable storage device of claim 16, wherein generating the plurality of printable characters representing the file includes converting at least a portion of the file to American Standard Code for Information Interchange (ASCII) characters.

18. The computer-readable storage device of claim 14, wherein the supervised training data includes the plurality of n-gram vectors for each file.

19. The computer-readable storage device of claim 18, further including multiple n-gram vectors indicating occurrences of pairs of printable characters.