Application execution control utilizing ensemble machine learning for discernment

• US 10,235,518 B2
• Filed: 02/06/2015
• Issued: 03/19/2019
• Est. Priority Date: 02/07/2014
• Status: Active Grant

• Current Assignee: Cylance Inc. (Blackberry Limited)
• Original Assignee: Cylance Inc. (Blackberry Limited)
• Inventors: Permeh, Ryan, Soeder, Derek A., Chisholm, Glenn, Russell, Braden, Golomb, Gary, Wolff, Matthew, McClure, Stuart
• Primary Examiner(s): Gergiso, Techane
• Application Number:
  US14/616,509
• Publication Number:
  US 20150227741A1
• Time in Patent Office: 1,502 Days
• CPC Class Codes: G06F 21/51 at application loading time, e.g. accepting, rejecting, starting or inhibiting executable software based on integrity or source reliability
• CPC Class Codes: G06F 21/52 during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06F 2221/033 Test or assess software
• CPC Class Codes: G06F 2221/2115 Third party
• CPC Class Codes: G06F 2221/2125 Just-in-time application of countermeasures, e.g., on-the-fly decryption, just-in-time obfuscation or de-obfuscation
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 63/12 Applying verification of the received information cryptographic mechanisms or cryptographic arrangements for data integrity or data verification H04L9/32
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002

Abstract

Described are techniques to enable computers to efficiently determine if they should run a program based on an immediate (i.e., real-time, etc.) analysis of the program. Such an approach leverages highly trained ensemble machine learning algorithms to create a real-time discernment on a combination of static and dynamic features collected from the program, the computer'"'"'s current environment, and external factors. Related apparatus, systems, techniques and articles are also described.

Claims

1. A method for implementation by one or more computer systems comprising:

receiving, from a feature collector, at least one feature from a plurality of possible features to enable a determination of whether to execute or continue to execute at least a portion of a program;

selecting, by a model collector, a machine learning model from an existing ensemble of machine learning models which can be used to discern at least the portion of the program, the selected machine learning model enabling a determination of whether to allow at least the portion of the program to execute or continue to execute based on whether such at least the portion of the program is deemed safe or unsafe;

determining, based on the selected machine learning model, whether to allow at least the portion of the program to execute or continue to execute;

allowing at least the portion of the program to execute or continue to execute, when the selected machine learning model determines that at least the portion of the program is allowed to execute or continue to execute; and

preventing at least the portion of the program from executing or continuing to execute, when the selected machine learning model determines that at least the portion of the program is not allowed to execute or continue to execute;

wherein selection of the machine learning model by the model collector is predicated on either which of the possible features are received from the feature collector or a current availability or scarcity of computing resources.

2. The method as in claim 1, wherein the selected machine learning model is trained using feature data derived from a plurality of different programs.

3. The method as in claim 1, wherein the selected machine learning model is trained using supervised learning.

4. The method as in claim 1, wherein the selected machine learning model is trained using unsupervised learning.

5. The method as in claim 1 further comprising:

collecting, by the feature collector, the received at least one feature.

6. The method as in claim 5, wherein the feature collector collects features at a pre-specified point in time.

7. The method as in claim 6, wherein the pre-specified point in time is at execution or at execution continuation of at least the portion of the program.

8. The method as in claim 1, wherein the received at least one feature comprises a combination of point in time measurements and ongoing measurements during execution or execution continuation of at least the portion of the program.

9. The method as in claim 1, wherein the received at least one feature comprises one or more operational features passively collected prior to execution or execution continuation of at least the portion of the program, and wherein the method further comprises:

storing the one or more operational features in a cache.

10. The method as in claim 1, wherein the received at least one feature comprises reputation of at least the portion of the program, contextual information, system state, system operating statistics, time-series data, at least a portion of existing programs, operating system details and/or run status of at least the portion of the program and configuration variables.

11. The method as in claim 1, wherein the received at least one feature comprises measurements of at least the portion of the program, structural elements of at least the portion of the program and/or contents of at least the portion of the program.

12. The method as in claim 1, wherein the received at least one feature comprises interactions with an operating system, subroutine executions, process state, execution statistics of at least the portion of the program or a system and/or an order of an occurrence of events associated with at least the portion of the program.

13. The method as in claim 1, wherein the received at least one feature is obtained from at least one remote database or other data source.

14. The method as in claim 1, wherein the received at least one feature takes a format that comprises binary, continuous and/or categorical.

15. The method as in claim 1, wherein one or more machine learning models of the ensemble of machine learning models comprises neural network models, support vector machine models, scorecard models, logistic regression models, Bayesian models and/or decision tree models.

16. The method as in claim 1, wherein the selecting of the machine learning model includes, at least, selecting two or more machine learning models.

17. The method as in claim 1, wherein the determining comprises generating a score characterizing a level of safety for executing or continuing to execute at least the portion of the program, wherein the generated score is used to determine whether to allow at least the portion of the program to execute or continue to execute.

18. The method as in claim 17, wherein the determining further comprises generating a confidence level for the generated score, wherein the generated confidence level is used to determine whether to allow at least the portion of the program to execute or continue to execute.

19. The method as in claim 1, wherein the preventing of at least the portion of the program from executing or continuing to execute comprises blocking at least the portion of the program from loading into memory.

20. The method as in claim 1 further comprising:

determining that a dynamic library associated with at least the portion of the program is unsafe;

wherein the preventing of at least the portion of the program from executing or continuing to execute comprises;

blocking the dynamic library associated with at least the portion of the program from loading into memory.

21. The method as in claim 1, wherein the preventing of at least the portion of the program from executing or continuing to execute comprises:

unloading a previously loaded module associated with at least the portion of the program.

22. The method as in claim 1, wherein the preventing of at least the portion of the program from executing or continuing to execute comprises:

disabling at least the portion of the program while it is running.

23. The method as in claim 1, wherein the preventing of at least the portion of the program from executing or continuing to execute comprises one or more of:

implementing constraints on at least the portion of the program prior to it being run or before it continues to run;

quarantining at least the portion of the program; and

deleting at least the portion of the program.

24. The method as in claim 1, wherein the preventing of at least the portion of the program from executing or continuing to execute comprises one or more of:

preventing at least the portion of the program from executing individual operations, modifying an access level of at least the portion of the program, selectively blocking attempted operations and preventing an attempted operation and instead causing an alternative operation.

25. The method of claim 1, wherein the received at least one feature comprises at least one operational feature that characterizes an operational environment of a system to execute at least the portion of the program.

26. The method of claim 1, wherein the received at least one feature comprises at least one dynamic feature that characterizes execution of at least the portion of the program.

27. The method of claim 1, wherein the received at least one feature comprises at least one external feature from a source external to a system to execute at least the portion of the program.

28. The method of claim 1, wherein the received at least one feature comprises at least one static feature that characterizes at least the portion of the program.

29. A system comprising:

at least one hardware data processor; and

memory storing instructions which, when executed by the at least one hardware data processor, result in operations comprising;

receiving a plurality of features from at least two difference to enable a determination of whether to execute or continue to execute at least a portion of a program based on whether such at least the portion of the program is deemed safe or unsafe;

selecting, based on the received plurality of features, a machine learning model from an existing ensemble of machine learning models which can be used to discern at least the portion of the program, the selected machine learning model enabling a determination of whether to allow at least the portion of the program to execute or continue to execute;

determining, based on the selected machine learning model, whether to allow at least the portion of the program to execute or continue to execute;

allowing at least the portion of the program to execute or continue to execute, when the selected machine learning model determines that at least the portion of the program is allowed to execute or continue to execute; and

preventing at least the portion of the program from executing or continuing to execute, when the selected machine learning model determines that at least the portion of the program is not allowed to execute or continue to execute;

wherein the preventing of at least the portion of the program from executing or continuing to execute comprises one or more of;

implementing constraints on at least the portion of the program prior to it being run or before it continues to run;

quarantining at least the portion of the program; and

deleting at least the portion of the program.

30. The system of claim 29, wherein at least one feature from the received plurality of features comprises reputation information about at least the portion of the program.

31. The system of claim 29, wherein at least one feature from the received plurality of features comprises measurements of at least the portion of the program or structural elements of at least the portion of the program.

32. The system of claim 29, wherein the selecting of a machine learning model includes at least selecting two machine learning models.

33. The system of claim 29, wherein the at least two sources are selected from a group consisting of:

operational features that relate to an operational environment of the system, static features that are associated with the program, dynamic features that relate to execution of the program, or external features that are extracted from a source other than the system executing the program.