System and method for detecting malicious code executed by virtual machine

• US 8,713,631 B1
• Filed: 02/14/2013
• Issued: 04/29/2014
• Est. Priority Date: 12/25/2012
• Status: Active Grant

• Current Assignee: Lab A.O. Kaspersky
• Original Assignee: Kaspersky Lab ZAO
• Inventors: Pavlyushchik, Mikhail A.
• Primary Examiner(s): Lipman, Jacob
• Application Number:
  US13/767,391
• Time in Patent Office: 439 Days
• Field of Search: 726/1, 726/23
• US Class Current: 726/1
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities

Abstract

Protection against a malicious set of program instructions (e.g., a malicious program) executable by a process virtual machine. The program instructions of process virtual machine are augmented to establish an exception monitoring module within the process virtual machine. When the process virtual machine executes a subject set of program instructions, the exception monitoring module detects a security policy violation exception occurring as a result. In response thereto, the exception monitoring module gathers context information representing circumstances surrounding the occurrence of the exception, and provides the context information for analysis of a presence of malicious code. The exception monitoring module determines, based on a result of the analysis, whether to permit further execution of the subject set of program instructions by the process virtual machine.

Claims

1. An automated computer-implemented method for protecting against a malicious set of program instructions that are executable by a process virtual machine, the process virtual machine comprising program instructions executable on a computer system having a hardware platform and an operating system, the method comprising:

augmenting, by an automated augmentation process executing on the computer system, the program instructions of the process virtual machine to establish an exception monitoring module within the process virtual machine;

executing, via the process virtual machine, the subject set of program instructions;

detecting, by the exception monitoring module, an exception occurring as a result of the execution of the subject set of program instructions, wherein the exception represents an occurrence of an event determined as having potential to violate a predefined security policy;

in response to a detection of an occurrence of the exception, gathering, by the exception monitoring module, context information from the process virtual machine, the context information representing circumstances surrounding the occurrence of the exception;

providing, by the exception monitoring module, the context information to be subjected to analysis for a presence of the malicious set of program instructions; and

determining, by the exception monitoring module, based on a result of the analysis, whether to permit further execution of the subject set of program instructions by the process virtual machine.

2. The method of claim 1, further comprising:

in response to a detection of an occurrence of the exception, stopping operation of the process virtual machine to prevent further execution of the subject set of program instructions; and

in response to a result of the analysis being indicative of an absence of malicious code from the subject set of program instructions, re-starting operation of the process virtual machine.

3. The method of claim 1, further comprising:

in response to a detection of an occurrence of the exception, stopping operation of the process virtual machine to prevent further execution of the subject set of program instructions; and

in response to a result of the analysis being indicative of a presence of malicious code in the subject set of program instructions, terminating operation of the process virtual machine.

4. The method of claim 1, wherein the augmenting is performed after initiation of execution of the program instructions of the process virtual machine.

5. The method of claim 1, wherein the augmenting is performed prior to initiation of execution of the program instructions of the process virtual machine.

6. The method of claim 1, wherein in the augmenting, specialized code for implementing the exception monitoring module is inserted into an existing security manager portion of the process virtual machine.

7. The method of claim 1, wherein the augmenting includes function overloading of one or more existing functions of the process virtual machine.

8. The method of claim 1, wherein in providing the context information, the circumstances surrounding the exception that are provided include information indicative of a cause of the exception.

9. The method of claim 1, wherein in providing the context information, the circumstances surrounding the exception that are provided include information indicative of events preceding the occurrence of a critical event identified as the exception.

10. The method of claim 1, further comprising:

receiving, by an analysis module executing, on the computer system, the context information; and

determining, by the analysis module, whether the context information is associated with malicious code.

11. The method of claim 10, wherein the determining whether the context information is associated with malicious code includes comparing the context information against a set of at least one predefined template representing malicious activity.

12. A system for protecting against malicious functionality of a subject set of program instructions, the system comprising:

a computer system having a hardware platform and an operating system executable on the hardware platform;

a process virtual machine module executable on the computer system that, when executed, forms a virtual execution environment in which the subject set of program instructions are executable; and

a control module executable on the computer system that, when executed, augments the process virtual machine to insert an exception monitoring module configured to;

detect an exception occurring as a result of execution of the subject set of program instructions, wherein the exception represents an occurrence of an event determined as having potential to violate a predefined security policy;

in response to a detection of an occurrence of the exception, gather context information from operation of the process virtual machine representing circumstances surrounding the occurrence of the exception;

provide the context information to be subjected to analysis for a presence of the malicious code; and

determine, based on a result of the analysis, whether to permit further execution of the subject set of program instructions by the process virtual machine.

13. The system of claim 12, wherein the process virtual machine is configured:

in response to a detection of an occurrence of the exception, cease operation to prevent further execution of the subject set of program instructions; and

to re-start the operation of the process virtual machine in response to a result of the analysis being indicative of an absence of malicious code from the subject set of program instructions.

14. The system of claim 12, wherein the process virtual machine is configured:

in response to a detection of an occurrence of the exception, cease operation to prevent further execution of the subject set of program instructions; and

to terminate operation of the process virtual machine in response to a result of the analysis being indicative of a presence of malicious code in the subject set of program instructions.

15. The system of claim 12, wherein the control module is configured to augment the process virtual machine in response to initiation of execution of the program instructions in the process virtual machine.

16. The system of claim 12, wherein the control module is configured to augment the process virtual machine prior to initiation of execution of the program instructions of the process virtual machine.

17. The system of claim 12, wherein the process virtual machine includes a security manager portion, and wherein the control module is configured to insert specialized code for implementing the exception monitoring module into the security manager portion of the process virtual machine.

18. The system of claim 12, wherein the control module is configured to augment the process virtual machine by function overloading one or more existing functions of the process virtual machine.

19. The system of claim 12, wherein the circumstances surrounding the exception include information indicative of a cause of the exception.

20. The system of claim 12, wherein the circumstances surrounding the exception include information indicative of events preceding the occurrence of a critical event identified as the exception.

21. The system of claim 12, further comprising:

an analysis module executable on the computer system, the analysis module configured to obtain the context information, and to determine whether the context information is associated with malicious code.

22. The system of claim 21, wherein the analysis module is configured to compare the context information against a set of at least one predefined template representing malicious activity.

23. An automated computer-implemented system for protecting against a malicious set of program instructions that are executable by a process virtual machine, the process virtual machine comprising program instructions executable on a hardware platform via an operating system, the system comprising:

means for augmenting, by an automated augmentation process executing on the computer system, the program instructions of the process virtual machine to establish an exception monitoring module within the process virtual machine;

means for detecting, by the exception monitoring module, an exception occurring as a result of execution of the subject set of program instructions via the process virtual machine, wherein the exception represents an occurrence of an event determined as having potential to violate a predefined security policy;

means for gathering, in response to a detection of an occurrence of the exception, context information from the process virtual machine, the context information representing circumstances surrounding the occurrence of the exception;

means for providing the context information for analysis to determine a presence of the malicious code; and

means for determining, based on a result of the analysis, whether to permit further execution of the subject set of program instructions by the process virtual machine.