Static feature extraction from structured files

• US 9,959,276 B2
• Filed: 02/12/2016
• Issued: 05/01/2018
• Est. Priority Date: 01/31/2014
• Status: Active Grant

• Current Assignee: Cylance Inc. (Blackberry Limited)
• Original Assignee: Cylance Inc. (Blackberry Limited)
• Inventors: Soeder, Derek A., Permeh, Ryan, Golomb, Gary, Wolff, Matthew
• Primary Examiner(s): Bullock, Jr., Lewis A
• Assistant Examiner(s): Gooray, Mark A
• Application Number:
  US15/043,276
• Publication Number:
  US 20160246800A1
• Time in Patent Office: 809 Days
• Field of Search: None
• CPC Class Codes: G06F 11/36 Preventing errors by testing or debugging software
• CPC Class Codes: G06F 11/3604 Software analysis for verifying properties of programs byte-code verification G06F9/44589
• CPC Class Codes: G06F 16/116 Details of conversion of file system types or formats
• CPC Class Codes: G06F 16/188 Virtual file systems
• CPC Class Codes: G06F 21/562 Static detection
• CPC Class Codes: G06F 21/563 by source code analysis
• CPC Class Codes: G06F 21/564 by virus signature recognition
• CPC Class Codes: G06F 21/565 by checking file integrity
• CPC Class Codes: G06F 40/205 Parsing

Abstract

Data is received or accessed that includes a structured file encapsulating data required by an execution environment to manage executable code wrapped within the structured file. Thereafter, code and data regions are iteratively identified in the structured file. Such identification is analyzed so that at least one feature can be extracted from the structured file. Related apparatus, systems, techniques and articles are also described.

Claims

1. A computer-implemented method comprising:

receiving and accessing a plurality of structured files;

parsing each structured file to discover corresponding code and data regions and to extract a plurality of corresponding code start points;

extracting, for each structured file, at least one feature from such structured a file by disassembling the code in the structured file using each of the plurality of corresponding code start points as a respective disassembly starting point and analyzing one or more code and data regions identified within the structured file, the extracting occurring statically while the structured file is not being executed, the features being at least one of (i) a first-order feature indicating whether a collection of import names is ordered lexicographically and being able to be derived into a higher-order feature, (ii) a checksum feature for a string of elements in the file compared to a checksum stored in a field in the file, or (iii) a Boolean feature that characterizes a set of timestamp fields from the file to represent whether or not the file relies upon various functionalities that did not exist at the time represented by the most recent time stamp;

providing the extracted features from each of the plurality of structured files to a machine learning model, to determine classification of the features and place them into a malicious or benign category, wherein the provision of the extracted features from one of the plurality of structured files, reduces subsequent misclassification of extracted features from the next one of the plurality structured files by the machine learning model.

2. A method as in claim 1, wherein the feature comprises a first order feature.

3. A method as in claim 1, further comprising processing the first order feature into a higher order feature.

4. A method as in claim 1, further comprising:

analyzing negative space within the file to extract an additional feature, wherein the negative space is different from the code and/or data regions identified within the file, the method further comprising providing the additional features to the machine learning model, the determining further comprising also analyzing the additional feature.

5. A method as in claim 4, wherein the negative space does not correspond to any identifiable code or data in the file.

6. A method as in claim 1, wherein the structured file encapsulates data required by an execution environment to manage executable code wrapped within the structured file.

7. A method as in claim 1, further comprising:

determining that the structured file is valid by examining at least one header within the structured file to determine whether it encapsulates a valid signature.

8. A method as in claim 1, wherein the extracting further comprises:

reading and parsing code and/or data in the file; and

iteratively discovering additional code and/or data.

9. A method as in claim 1, wherein the disassembling of the code comprises discovering data references;

and the method further comprises;

adding the discovered data references to a queue of data to be read and parsed.

10. A method as in claim 1, wherein the extracting further comprises maintaining both a hierarchy and an order of the file.

11. A method as in claim 1, further comprising:

transforming the feature extracted from the file.

12. A method as in claim 11, wherein the transforming comprises at least one of sanitizing, truncating, and escaping or encoding data in an unambiguously reversible way.

13. A method as in claim 1, wherein the classification comprises a computer security classification.

14. A method as in claim 1, wherein the machine learning model comprises a classification model.

15. A system comprising:

at least one processor; and

at least one memory including instructions which, when executed by the at least one processor, result in the at least one processor performing operations comprising;

receiving and accessing a plurality of structured files;

parsing each structured file to discover corresponding code and data regions and to extract a plurality of corresponding code start points;

extracting, for each structured file, at least one feature from such structured a file by disassembling the code in the structured file using each of the plurality of corresponding code start points as a respective disassembly starting point and analyzing one or more code and data regions identified within the structured file the extracting occurring statically while the structured file is not being executed, the features being at least one of (i) a first-order feature indicating whether a collection of import names is ordered lexicographically and being able to be derived into a higher-order feature, (ii) a checksum feature for a string of elements in the file compared to a checksum stored in a field in the file, or (iii) a Boolean feature that characterizes a set of timestamp fields from the file to represent whether or not the file relies upon various functionalities that did not exist at the time represented by the most recent time stamp;

providing the extracted features from each of the plurality of structured files to a machine learning model, to determine classification of the features and place them into a malicious or benign category, wherein the provision of the extracted features from one of the plurality of structured files, reduces subsequent misclassification of extracted features from the next one of the plurality structured files by the machine learning model.

16. A system as in claim 15, wherein the operations further comprise:

analyzing negative space within the file to extract an additional feature, wherein the negative space is different from the code and/or data regions identified within the file; and

providing the additional features to the machine learning model, the determining further comprising also analyzing the additional feature.

17. A system as in claim 15, wherein the feature comprises a first order feature.

18. A system as in claim 15, wherein the operations further comprise:

processing the first order feature into a higher order feature.

19. A system as in claim 16, wherein the negative space does not correspond to any identifiable code or data in the file.

20. A system as in claim 15, wherein the structured file encapsulates data required by an execution environment to manage executable code wrapped within the structured file.

21. A system as in claim 15, wherein the operations further comprise:

determining that the structured file is valid by examining at least one header within the structured file to determine whether it encapsulates a valid signature.

22. A system as in claim 15, wherein the extracting further comprises:

reading and parsing code and/or data in the file; and

iteratively discovering additional code and/or data.

23. A system as in claim 15, wherein the disassembling of the code comprises discovering data references;

and the operations further comprise;

adding the discovered data references to a queue of data to be read and parsed.

24. A system as in claim 15, wherein the extracting further comprises maintaining both a hierarchy and an order of the file.

25. A system as in claim 15, wherein the operations further comprise:

transforming the feature extracted from the file.

26. A system as in claim 25, wherein the transforming comprises at least one of sanitizing, truncating, and escaping or encoding data in an unambiguously reversible way.

27. A system as in claim 15, wherein the classification comprises a computer security classification.

28. A system as in claim 15, wherein the machine learning model comprises a classification model.

29. A non-transitory computer-readable storage medium including instructions, which when executed by at least one processor, cause at least one processor to perform operations comprising:

receiving and accessing a plurality of structured files;

parsing each structured file to discover corresponding code and data regions and to extract a plurality of corresponding code start points;

extracting, for each structured file, at least one feature from such structured file by disassembling the code in the structured file using each of the plurality of corresponding code start points as a respective disassembly starting point and analyzing one or more code and data regions identified within the structured file the extracting occurring statically while the structured file is not being executed, the features being at least one of (i) a first-order feature indicating whether a collection of import names is ordered lexicographically and being able to be derived into a higher-order feature, (ii) a checksum feature for a string of elements in the file compared to a checksum stored in a field in the file, or (iii) a Boolean feature that characterizes a set of timestamp fields from the file to represent whether or not the file relies upon various functionalities that did not exist at the time represented by the most recent timestamp;

providing the extracted features from each of the plurality of structured files to a machine learning model, to determine classification of the features and place them into a malicious or benign category, wherein the provision of the extracted features from one of the plurality of structured files, reduces subsequent misclassification of extracted features from the next one of the plurality structured files by the machine learning model.

30. A non-transitory computer-readable storage medium as in claim 29, wherein the operations further comprise:

analyzing negative space within the file to extract an additional feature, wherein the negative space is different from the code and/or data regions identified within the file; and

providing the additional features to the machine learning model, the determining further comprising also analyzing the additional feature.

31. A non-transitory computer-readable storage medium as in claim 29, wherein the feature comprises a first order feature.

32. A non-transitory computer-readable storage medium as in claim 29, wherein the operations further comprise:

processing the first order feature into a higher order feature.

33. A non-transitory computer-readable storage medium as in claim 30, wherein the negative space does not correspond to any identifiable code or data in the file.

34. A non-transitory computer-readable storage medium as in claim 29, wherein the structured file encapsulates data required by an execution environment to manage executable code wrapped within the structured file.

35. A non-transitory computer-readable storage medium as in claim 29, wherein the operations further comprise:

determining that the structured file is valid by examining at least one header within the structured file to determine whether it encapsulates a valid signature.

36. A non-transitory computer-readable storage medium as in claim 29, wherein the extracting further comprises:

reading and parsing code and/or data in the file; and

iteratively discovering additional code and/or data.

37. A non-transitory computer-readable storage medium as in claim 29, wherein the disassembling of the code comprises discovering data references;

and the operations further comprise;

adding the discovered data references to a queue of data to be read and parsed.

38. A non-transitory computer-readable storage medium as in claim 29, wherein the extracting further comprises maintaining both a hierarchy and an order of the file.

39. A non-transitory computer-readable storage medium as in claim 29, wherein the operations further comprise:

transforming the feature extracted from the file.

40. A non-transitory computer-readable storage medium as in claim 39, wherein the transforming comprises at least one of sanitizing, truncating, and escaping or encoding data in an unambiguously reversible way.

41. A non-transitory computer-readable storage medium as in claim 29, wherein the classification comprises a computer security classification.

42. A non-transitory computer-readable storage medium as in claim 29, wherein the machine learning model comprises a classification model.