Selective system call monitoring

• US 9,690,606 B1
• Filed: 03/25/2015
• Issued: 06/27/2017
• Est. Priority Date: 03/25/2015
• Status: Active Grant

• Current Assignee: Musarubra US LLC (Musarubra US SellCo LLC)
• Original Assignee: FireEye, Inc. (Alphabet Inc.)
• Inventors: Ha, Phung-Te, Xu, Wei
• Primary Examiner(s): Thiaw, Catherine
• Application Number:
  US14/668,601
• Time in Patent Office: 825 Days
• CPC Class Codes: G06F 2009/45587 Isolation or security of virtual machine instances
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06F 2221/034 Test or assess a computer or a system
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

According to one embodiment of the invention, a computerized method is described for improved efficiency in malware detection. The method comprises detecting a system call initiated by a virtual machine and determining a class assigned to the detected system call. In response to determining that the system call is associated with a first class of system calls, providing information associated with the system call to virtualized device hardware. In contrast, in response to determining that the system call is associated with a second class of system calls, which is different from the first class of system calls, the virtual machine resumes virtual processing of an object without providing information to the virtualized device hardware.

Claims

1. A computerized method performed by a network appliance, comprising:

determining, by the network appliance including one or more hardware processors, whether a detected system call, which is generated by a process that is executing an object within a virtual machine, belongs to a first class of system calls by at least

halting operations by the virtual machine in response to the detected system call,

determining that there exists a prescribed level of likelihood that the process is associated with a malicious attack that identifies the process is suspicious,

responsive to determining that the process is not suspicious, comparing the identifier for the detected system call to each identifier for a first plurality of system calls that are part of the first class of system calls, and subsequently determining that the process is associated with a malicious attack by determining that the detected system call belongs to the first class of system calls upon successfully comparing the identifier for the detected system call to a first identifier for one of the first class of system calls; and

providing information associated with the system call to virtualized device hardware in response to determining that the system call is associated with the first class of system calls.

2. The computerized method of claim 1, wherein prior to determining whether the detected system call initiated by the virtual machine belongs to the first class of system calls, the method further comprises configuring a virtual machine monitor (VMM) to detect system calls initiated by the virtual machine.

3. The computerized method of claim 1, wherein the determining whether the detected system call belongs to the first class of system calls comprises:

accessing one or more data stores within the virtualized device hardware while the virtual machine is halted from operation to determine the identifier for the detected system call; and

comparing the identifier for the detected system call to each identifier for a plurality of system calls that are part of the first class of system calls.

4. The computerized method of claim 3, wherein prior to determining whether the detected system call initiated by the virtual machine belongs to the first class of system calls, the method further comprises configuring a virtual machine monitor (VMM) to detect system calls initiated by the virtual machine.

5. The computerized method of claim 1, further comprising

in lieu of providing information associated with the system call to the virtualized device hardware, resuming processing of the object by the process running on the virtual machine in response to determining that the system call differs from the first class of system calls.

6. The computerized method of claim 5, wherein the first class of system calls includes system-wide system calls and the second class of system calls includes process-specific system calls.

7. The computerized method of claim 6, wherein the second class of system calls includes a first subset of system calls for the process and a second subset of system calls for a second process different than the process.

8. The computerized method of claim 1, wherein the virtualized hardware device includes (i) one or more registers of a virtual processor or (ii) a virtualized memory.

9. The computerized method of claim 1, wherein the first class of system calls includes system-wide system calls that represent selected system calls that occur during malicious attacks, but are less processing intensive and/or less frequently initiated than other types of system calls.

10. The computerized method of claim 1, wherein the first class of system calls includes a system call that initiates a second process different from the process.

11. The computerized method of claim 1, wherein the process is a software component initiating the detected system call during execution of the object, the software component including (i) an application or (ii) an operating system.

12. The computerized method of claim 11, wherein the application includes a web browser.

13. The computerized method of claim 1, wherein the process includes the object that initiates the detected system call.

14. A computerized method for determining whether a detected system call that is generated by a process executing an object on a virtual machine is potentially associated with a malicious attack, the method comprising:

halting operations by the virtual machine executed by a network appliance including one or more hardware processors, in response to detecting the system call;

determining an identifier for the system call;

determining whether a prescribed level of likelihood exists that the process is associated with a malicious attack that identifies the process is suspicious;

responsive to determining that the process is not suspicious, comparing the identifier for the system call to each identifier for a first plurality of system calls that are part of a first class of system calls and a second class of system calls different than the first class of system calls; and

determining that the system call belongs to one of the first class of system calls and the second class of system calls upon successfully comparing the identifier for the system call to a first identifier for one of the first class of system calls and the second class of system calls.

15. The computerized method of claim 14 further comprising:

providing information associated with the system call to virtualized device hardware in response to determining that the system call is associated with either the first class of system calls or the second class of system calls.

16. The computerized method of claim 15, wherein the virtualized device hardware includes a virtual memory from which parameters needed to complete execution of one or more instructions that are part of the system call.

17. The computerized method of claim 15, wherein the determining whether the prescribed level of likelihood exists, the comparing of the identifier for the system call, and the determining that the system call belongs to one of the first class of system calls and the second class of system calls is performed by a virtual machine monitor.

18. The computerized method of claim 15, wherein the determining whether the prescribed level of likelihood exists, the comparing of the identifier for the system call, and the determining that the system call belongs to one of the first class of system calls and the second class of system calls is performed by a virtual machine memory inspection (VMMI) logic that is part of a virtual machine monitor.

19. The computerized method of claim 14, wherein the first class of system calls includes a system call that initiates a second process different from the process.

20. The computerized method of claim 14, wherein the process is a software component initiating the detected system call during execution of the object, the software component including (i) an application or (ii) an operating system.

21. The computerized method of claim 20, wherein the application includes a web browser.

22. The computerized method of claim 14, wherein the process includes the object that initiates the detected system call.

23. A computerized method comprising:

detecting a system call initiated by a virtual machine executed by a network appliance including one or more hardware processors, the detecting of the system call comprises

halting operations by the virtual machine in response to detecting the system call,

determining an identifier for the system call,

determining that there exists a prescribed level of likelihood that a process running on the virtual machine and processing an object is associated with a malicious attack to identify that the process is suspicious,

responsive to determining that the process is not suspicious, comparing the identifier for the system call to each identifier for a first plurality of system calls that are part of a first class of system calls, and

determining that the system call belongs to the first class of system calls upon successfully comparing the identifier for the system call to a first identifier for one of the first plurality of system calls;

providing information associated with the system call to virtualized device hardware in response to determining that the system call is associated with the first class of system calls; and

resuming processing of an object by the virtual machine without providing information associated with the system call to the virtualized device hardware in response to determining that the system call is associated with a second class of system calls that is different from the first class of system calls.

24. The computerized method of claim 23, wherein the detecting of the system call comprises configuring a virtual machine monitor (VMM) to detect system calls initiated by the virtual machine.

25. The computerized method of claim 23, wherein the determining whether the system call belongs to the first class of system calls comprises:

accessing one or more data stores within the virtualized device hardware while operations of the virtual machine are halted to determine an identifier for the system call; and

comparing the identifier for the system call to each identifier for a plurality of system calls that are part of the first class of system calls.

26. The computerized method of claim 23, wherein prior to determining the class assigned to the system call, the method further comprises configuring a virtual machine monitor (VMM) to detect system calls initiated by the virtual machine.

27. The computerized method of claim 23, wherein the first class of system calls includes system-wide system calls and the second class of system calls includes process-specific system calls.

28. The computerized method of claim 23, wherein the second class of system calls includes a first subset of system calls for the process and a second subset of system calls for a second process different than the process.

29. An apparatus comprising:

one or more processors; and

a memory coupled to the one or more processors, the memory comprises software that, when executed by the one or more processors, generates

(1) a virtual machine including a process that executes an object and initiates a system call during processing of the object, and

(2) a virtual machine monitor that (a) detects the system call initiated by the virtual machine, (b) determines a class assigned to the detected system call by at least comparing an identifier for the detected system call to each identifier for a first plurality of system calls that are part of a first class of system calls in response to failing to determine that the process is suspicious based on activities of the process failing to suggest that the process is associated with a malicious attack, (c) provides information associated with the system call to virtualized device hardware in response to determining that the system call is associated with the first class of system calls, and (d) resumes processing of the object by the virtual machine providing the information to the virtual device hardware in response to determining that the system call is associated with a second class of system calls that is different from the first class of system calls.

30. The apparatus of claim 29, wherein the determining of the class assigned to the detected system call by the virtual machine monitor comprises:

halting operations by the virtual machine in response to detecting the system call;

accessing one or more data stores within the virtualized device hardware while operations of the virtual machine are halted to determine the identifier for the detected system call; and

comparing the identifier for the detected system call to each identifier for a plurality of system calls that are part of the first class of system calls.

31. The apparatus of claim 29, wherein the determining of the class assigned to the detected system call by the virtual machine monitor further comprises:

halting operations by the virtual machine in response to detecting the system call;

accessing one or more data stores within the virtualized device hardware while operations of the virtual machine are halted to determine the identifier for the detected system call;

determining that the process is suspicious when there exists a prescribed level of likelihood that the process running on the virtual machine is associated with a malicious attack;

responsive to determining that the process is not suspicious, comparing the identifier for the detected system call to each identifier for the first plurality of system calls that are part of the first class of system calls; and

determining that the detected system call belongs to the first class of system calls upon successfully comparing the identifier for the detected system call to a first identifier for one of the first class of system calls.

32. The apparatus of claim 29, wherein the first class of system calls includes system-wide system calls and the second class of system calls includes process-specific system calls that are different from system-wide system calls.

33. The apparatus of claim 29, wherein the first class of system calls includes a system call that initiates a second process different from the process.

34. The apparatus of claim 29, wherein the process running on the virtual machine is a software component that initiates the detected system call, the software component including (i) an application or (ii) an operating system.

35. The apparatus of claim 34, wherein the application includes a web browser.

36. The apparatus of claim 29, wherein the process running on the virtual machine includes the object that initiates the detected system call.

37. A non-transitory computer readable medium that includes software that, when processed by one or more processor, performs operations comprising:

detecting a system call initiated by a virtual machine that is included as part of the software and is being executed by the one or more processors;

responsive to determining that the system call is not malicious, determining a class assigned to the detected system call by at least comparing an identifier for the detected system call to each identifier for a first plurality of system calls that are part of a first class of system calls;

providing information associated with the system call to virtualized device hardware in response to determining that the system call is associated with the first class of system calls; and

resuming processing of an object by the virtual machine without providing information associated with the system call to the virtualized device hardware in response to determining that the system call is associated with a second class of system calls that is different from the first class of system calls.