Software assurance system for runtime environments

• US 10,558,809 B1
• Filed: 04/12/2017
• Issued: 02/11/2020
• Est. Priority Date: 04/12/2017
• Status: Active Grant

• Current Assignee: Architecture Technology Corporation
• Original Assignee: Architecture Technology Corporation
• Inventors: Joyce, Robert A., Donovan, Matthew P.
• Primary Examiner(s): Lagor, Alexander
• Assistant Examiner(s): Tran, Vu V
• Application Number:
  US15/485,784
• Time in Patent Office: 1,035 Days
• CPC Class Codes: G06F 21/52 during program execution, e.g. stack integrity ; Preventing unwanted data erasure; Buffer overflow
• CPC Class Codes: G06F 21/577 Assessing vulnerabilities and evaluating computer system security
• CPC Class Codes: G06F 2221/033 Test or assess software
• CPC Class Codes: G06F 2221/034 Test or assess a computer or a system
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/046 Forward inferencing; Production systems
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]

Abstract

An example method includes monitoring execution of one or more applications on a runtime computing system that includes a plurality of processing units, receiving, from the runtime computing system during execution of the applications, monitoring information that includes at least one of function call data or application programming interface call data associated with operations performed by the plurality of processing units during execution of the applications, importing the monitoring information into a risk model, analyzing the monitoring information within the risk model to determine one or more potential vulnerabilities and one or more impacts of the one or more vulnerabilities in the runtime computing system, and outputting, for display in a graphical user interface, a graphical representation of the one or more potential vulnerabilities and the one or more impacts within the risk model.

Claims

1. A method comprising:

monitoring, by an analysis computing system, execution of one or more applications on a runtime computing system, wherein the runtime computing system includes a plurality of processing units that perform one or more operations during execution of the one or more applications;

during execution of the one or more applications on the runtime computing system, receiving, by the analysis computing system and from the runtime computing system, monitoring information that includes at least one of function call data or application programming interface call data associated with the one or more operations performed by the plurality of processing units during execution of the one or more applications, wherein the at least one of the function call data or the application programming interface call data comprises at least one ordered sequence of a plurality of function calls or application programming interface calls that are each intercepted by at least one function hook or application programming interface hook during execution of the one or more applications on the runtime computing system;

importing, by the analysis computing system, the monitoring information into a risk model;

analyzing, by the analysis computing system, the monitoring information within the risk model to determine one or more potential vulnerabilities and one or more potential impacts of the one or more potential vulnerabilities in the runtime computing system, wherein the one or more potential vulnerabilities are associated with execution of the one or more applications on the runtime computing system, and wherein the one or more potential vulnerabilities are further associated with at least one unexpected call sequence or unexpected call stack associated with the at least one ordered sequence of the plurality of function calls or application programming interface calls; and

outputting, by the analysis computing system and for display in a graphical user interface, a graphical representation of the one or more potential vulnerabilities and the one or more potential impacts in the risk model.

2. The method of claim 1,

wherein the risk model comprises a hierarchical risk model having a tree structure, wherein hierarchical risk model includes a plurality of nodes comprising at least one root node and one or more sub-nodes, the at least one root node representing at least one higher-level operation associated with execution of the one or more applications, and the one or more sub-nodes each representing a respective lower-level operation associated with execution of the one or more applications,

wherein analyzing the monitoring information within the hierarchical risk model to determine the one or more potential vulnerabilities and the one or more potential impacts comprises evaluating the monitoring information within the hierarchical risk model to determine at least one node of the plurality of nodes that is associated with the one or more potential vulnerabilities and the one or more potential impacts, and

wherein outputting the graphical representation of the one or more potential vulnerabilities and the one or more potential impacts in the risk model comprises graphically emphasizing the at least one node of the hierarchical risk model.

3. The method of claim 2, wherein graphically emphasizing the at least one node of the hierarchical risk model comprises graphically emphasizing the at least one node based at least on a respective severity of each of the one or more vulnerabilities.

4. The method of claim 2, wherein the tree structure comprises an attack tree, wherein the at least one root node represents at least one higher-level attack goal of one or more attacks performed during execution of the one or more applications, and wherein the one or more sub-nodes each represents a respective lower-level attack sub-goal of the one or more attacks.

5. The method of claim 1, wherein the at least one of the function hook or the application programming interface hook intercepts one or more library calls and captures data associated with at least one of memory usage, system call invocation information, an execution state, or stack trace information associated with execution of the one or more applications.

6. The method of claim 1, wherein analyzing the monitoring information within the risk model to determine one or more potential vulnerabilities and the one or more potential impacts further comprises:

utilizing, by the analysis computing system, one or more machine learning models to identify a respective likelihood for occurrence of each of the one or more potential vulnerabilities that are associated with execution of the one or more applications, wherein the machine learning models are trained based on previously identified categories of function and application programming interface calls from prior execution of other applications.

7. The method of claim 6, wherein utilizing the one or more machine learning models comprises identifying, by the analysis computing system, based on at least one of the one or more potential vulnerabilities, a group of instructions of the one or more applications that are determined as potentially malicious.

8. The method of claim 6, wherein the previously identified categories of function and application programming interface calls comprise a first category of typical call flows and a second category of atypical call flows.

9. The method of claim 1, wherein analyzing the monitoring information within the risk model to determine the one or more potential vulnerabilities and the one or more potential impacts further comprises determining one or more test results, included in the monitoring information, indicating a failure of one or more tests performed during execution of the one or more applications.

10. The method of claim 1, further comprising:

receiving, by the analysis computing system, additional information associated with the runtime computing system; and

combining, by the analysis computing system, the monitoring information with the additional information in the risk model,

wherein analyzing the monitoring information further comprises analyzing, by the analysis computing system, the monitoring information combined with the additional information within the risk model to determine the one or more potential vulnerabilities and the one or more potential impacts of the one or more potential vulnerabilities in the runtime computing system.

11. The method of claim 10, wherein the additional information comprises one or more of:

source code information associated with an analysis of source code for at least one of an operating system or the one or more applications of the runtime computing system;

system-level state information associated with a state of at least one of the operating system or the one or more applications of the runtime computing system;

platform configuration information associated with a configuration of the runtime computing system;

or

probe information associated with one or more external interfaces provided by the runtime computing system.

12. The method of claim 1, further comprising importing, by the analysis computing system, knowledge base information into the risk model for use in determining the one or more potential vulnerabilities associated with execution of the one or more applications, wherein the knowledge base information comprises information associated with previously identified vulnerabilities for other runtime computing systems.

13. The method of claim 1, wherein the plurality of processing units of the runtime computing system includes at least one graphics processing unit that performs at least one of the one or more operations on the runtime computing system during execution of the one or more applications.

14. The method of claim 1, wherein the one or more potential impacts of the one or more potential vulnerabilities comprise at least one of (1) an impact of information confidentiality during execution of the one or more applications on the runtime computing system, (2) an impact of information integrity during execution of the one or more applications on the runtime computing system, or (3) an impact of information availability during execution of the one or more applications on the runtime computing system.

15. The method of claim 1, wherein the analysis computing system is different from the runtime computing system.

16. A computing system, comprising:

one or more processors; and

a non-transitory computer-readable storage medium storing instructions that, when executed, cause the one or more processors to;

monitor execution of one or more applications on a runtime computing system, wherein the runtime computing system includes a plurality of processing units that perform one or more operations during execution of the one or more applications;

during execution of the one or more applications on the runtime computing system, receive, from the runtime computing system, monitoring information that includes at least one of function call data or application programming interface call data associated with the one or more operations performed by the plurality of processing units during execution of the one or more applications, wherein the at least one of the function call data or the application programming interface call data comprises at least one ordered sequence of a plurality of function calls or application programming interface calls that are each intercepted by at least one function hook or application programming interface hook during execution of the one or more applications on the runtime computing system;

import the monitoring information into a risk model;

analyze the monitoring information within the risk model to determine one or more potential vulnerabilities and one or more potential impacts of the one or more potential vulnerabilities in the runtime computing system, wherein the one or more potential vulnerabilities are associated with execution of the one or more applications on the runtime computing system, and wherein the one or more potential vulnerabilities are further associated with at least one unexpected call sequence or unexpected call stack associated with the at least one ordered sequence of the plurality of function calls or application programming interface calls; and

output, for display in a graphical user interface, a graphical representation of the one or more potential vulnerabilities and the one or more potential impacts within the risk model.

17. The computing system of claim 16,

wherein the risk model comprises a hierarchical risk model having a tree structure, wherein hierarchical risk model includes a plurality of nodes comprising at least one root node and one or more sub-nodes, the at least one root node representing at least one higher-level operation associated with execution of the one or more applications, and the one or more sub-nodes each representing a respective lower-level operation associated with execution of the one or more applications,

wherein the instructions that cause the one or more processors to analyze the monitoring information within the hierarchical risk model to determine the one or more potential vulnerabilities and the one or more potential impacts further cause the one or more processors to evaluate the monitoring information within the hierarchical risk model to determine at least one node of the plurality of nodes that is associated with the one or more potential vulnerabilities and the one or more potential impacts, and

wherein the instructions that cause the one or more processors to output the graphical representation of the one or more potential vulnerabilities and the one or more potential impacts in the risk model further cause the one or more processors to graphically emphasize the at least one node of the hierarchical risk model.

18. The computing system of claim 16, wherein the instructions that cause the one or more processors to analyze the monitoring information within the risk model to determine one or more potential vulnerabilities and the one or more potential impacts further cause the one or more processors to:

utilize one or more machine learning models to identify a respective likelihood for occurrence of each of the one or more potential vulnerabilities that are associated with execution of the one or more applications, wherein the machine learning models are trained based on previously identified categories of function and application programming interface calls from prior execution of other applications.

19. The computing system of claim 16, wherein the instructions further cause the one or more processors to:

receive additional information associated with the runtime computing system; and

combine the monitoring information with the additional information in the risk model,

wherein the instructions that cause the one or more processors to analyze the monitoring information further cause the one or more processors to analyze the monitoring information combined with the additional information within the risk model to determine the one or more potential vulnerabilities and the one or more potential impacts of the one or more potential vulnerabilities in the runtime computing system.

20. The computing system of claim 19, wherein the additional information comprises one or more of:

source code information associated with an analysis of source code for at least one of an operating system or the one or more applications of the runtime computing system;

system-level state information associated with a state of at least one of the operating system or the one or more applications of the runtime computing system;

platform configuration information associated with a configuration of the runtime computing system;

or

probe information associated with one or more external interfaces provided by the runtime computing system.

21. The computing system of claim 16, wherein the one or more potential impacts of the one or more potential vulnerabilities comprise at least one of (1) an impact of information confidentiality during execution of the one or more applications on the runtime computing system, (2) an impact of information integrity during execution of the one or more applications on the runtime computing system, or (3) an impact of information availability during execution of the one or more applications on the runtime computing system.

22. The computing system of claim 16, wherein the at least one of the function hook or the application programming interface hook intercepts one or more library calls and captures data associated with at least one of memory usage, system call invocation information, an execution state, or stack trace information associated with execution of the one or more applications.

23. The computing system of claim 16, wherein the plurality of processing units of the runtime computing system includes at least one graphics processing unit that performs at least one of the one or more operations on the runtime computing system during execution of the one or more applications.

24. A non-transitory computer-readable storage medium storing instructions that, when executed, cause a computing system to perform operations comprising:

monitoring execution of one or more applications on a runtime computing system, wherein the runtime computing system includes a plurality of processing units that perform one or more operations during execution of the one or more applications;

during execution of the one or more applications on the runtime computing system, receiving, from the runtime computing system, monitoring information that includes at least one of function call data or application programming interface call data associated with the one or more operations performed by the plurality of processing units during execution of the one or more applications, wherein the at least one of the function call data or the application programming interface call data comprises at least one ordered sequence of a plurality of function calls or application programming interface calls that are each intercepted by at least one function hook or application programming interface hook during execution of the one or more applications on the runtime computing system;

importing the monitoring information into a risk model;

analyzing the monitoring information within the risk model to determine one or more potential vulnerabilities and one or more potential impacts of the one or more potential vulnerabilities in the runtime computing system, wherein the one or more potential vulnerabilities are associated with execution of the one or more applications on the runtime computing system, and wherein the one or more potential vulnerabilities are further associated with at least one unexpected call sequence or unexpected call stack associated with the at least one ordered sequence of the plurality of function calls or application programming interface calls; and

outputting, for display in a graphical user interface, a graphical representation of the one or more potential vulnerabilities and the one or more potential impacts within the risk model.

25. The non-transitory computer-readable storage medium of claim 24,

wherein the risk model comprises a hierarchical risk model having a tree structure, wherein hierarchical risk model includes a plurality of nodes comprising at least one root node and one or more sub-nodes, the at least one root node representing at least one higher-level operation associated with execution of the one or more applications, and the one or more sub-nodes each representing a respective lower-level operation associated with execution of the one or more applications,

wherein analyzing the monitoring information within the hierarchical risk model to determine the one or more potential vulnerabilities and the one or more potential impacts comprises evaluating the monitoring information within the hierarchical risk model to determine at least one node of the plurality of nodes that is associated with the one or more potential vulnerabilities and the one or more potential impacts, and

wherein outputting the graphical representation of the one or more potential vulnerabilities and the one or more potential impacts in the risk model comprises graphically emphasizing the at least one node of the hierarchical risk model.

26. The non-transitory computer-readable storage medium of claim 24, wherein analyzing the monitoring information within the risk model to determine one or more potential vulnerabilities and the one or more potential impacts further comprises:

utilizing one or more machine learning models to identify a respective likelihood for occurrence of each of the one or more potential vulnerabilities that are associated with execution of the one or more applications, wherein the machine learning models are trained based on previously identified categories of function and application programming interface calls from prior execution of other applications.

27. The non-transitory computer-readable storage medium of claim 26, wherein utilizing the one or more machine learning models comprises identifying based on at least one of the one or more potential vulnerabilities, a group of instructions of the one or more applications that are determined as potentially malicious.

28. The non-transitory computer-readable storage medium of claim 24, wherein the one or more potential impacts of the one or more potential vulnerabilities comprise at least one of (1) an impact of information confidentiality during execution of the one or more applications on the runtime computing system, (2) an impact of information integrity during execution of the one or more applications on the runtime computing system, or (3) an impact of information availability during execution of the one or more applications on the runtime computing system.

29. The non-transitory computer-readable storage medium of claim 24, wherein the at least one of the function hook or the application programming interface hook intercepts one or more library calls and captures data associated with at least one of memory usage, system call invocation information, an execution state, or stack trace information associated with execution of the one or more applications.