System and method for detecting malicious traffic using a virtual machine configured with a select software environment

• US 9,356,944 B1
• Filed: 06/28/2013
• Issued: 05/31/2016
• Est. Priority Date: 04/01/2004
• Status: Active Grant

• Current Assignee: Fireeye Security Holdings US LLC
• Original Assignee: FireEye, Inc.
• Inventors: Aziz, Ashar
• Primary Examiner(s): Traore, Fatoumata
• Assistant Examiner(s): Dolly, Kendall
• Application Number:
  US13/931,633
• Time in Patent Office: 1,068 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2009/45587 Isolation or security of virtual machine instances
• CPC Class Codes: G06F 2009/45591 Monitoring or debugging support
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: H04L 63/0245 Filtering by information in the payload
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms
• CPC Class Codes: H04L 63/1458 Denial of Service
• CPC Class Codes: H04L 63/1491 using deception as countermeasure, e.g. honeypots, honeynets, decoys or entrapment

Abstract

The system comprises a traffic analysis device in communication with a network device. The traffic analysis device can analyze network traffic received over a communication network and duplicate at least select network communications within the network traffic having characteristics associated with malicious traffic when the network communications are determined through heuristic analysis to satisfy a heuristic threshold. The network device comprises a controller in communication with one or more virtual machines that are configured to (i) receive the duplicated network communications from the traffic analysis device, (ii) monitor a behavior of a first virtual machine of the one or more virtual machines in response to processing of the duplicated network communications within the first virtual machine, (iii) identify an anomalous behavior as an unexpected occurrence in the monitored behavior, and (iv) determine, based on the identified anomalous behavior, the presence of the malicious traffic in the duplicated network communications.

Claims

1. A network device comprising:

a plurality of virtual machines based on one or more software modules stored within a memory storage device; and

a hardware controller, operating with a first virtual machine of the plurality of virtual machines to

(i) monitor one or more behaviors of at least the first virtual machine,

(ii) accelerate one or more activities in the first virtual machine to detect a time-delayed malware that performs one or more operations to avoid detection, the accelerating of the one or more activities comprises (a) intercepting one or more time-sensitive system calls and (b) modifying the one or more time-sensitive system calls or corresponding one or more responses to the one or more of the time-sensitive system calls,

(iii) identify at least one behavior of the one or more monitored behaviors as an anomalous behavior, and

(iv) detect, based at least in part on the identified at least one anomalous behavior, the time-delayed malware in the first virtual machine.

2. The network device of claim 1, wherein the hardware controller operating with the first virtual machine to accelerate the one or more activities by reducing a time parameter of a sleep system call being one of the one or more time-sensitive system calls to reduce execution time of the sleep system call.

3. The network device of claim 1, wherein the hardware controller operating with the first virtual machine to accelerate the one or more activities by modifying a response to a time-of-day system call being one of the one or more time-sensitive system calls, the response to the time-of-day system call by the first virtual machine is to specify a future time.

4. The network device of claim 1, wherein the hardware controller includes an intrusion detection system that is configured to generate an intrusion alert in response to detecting the time-delayed malware.

5. The network device of claim 1, wherein the time-delayed malware comprises one or more computer worms.

6. The network device of claim 1, wherein the hardware controller operating with the first virtual machine to accelerate one or more activities in the first virtual machine to reduce a time for detecting the time-delayed malware.

7. The network device of claim 1, wherein the one or more time-sensitive system calls includes a call to be serviced by a response from an operating system.

8. The network device of claim 7, wherein the one or more time-sensitive system calls are generated by the time-delayed malware.

9. The network device of claim 1, wherein the hardware controller operating with the first virtual machine for accelerating the one or more activities in the first virtual machine prior to the hardware controller detecting the time-delayed malware in the first virtual machine.

10. The network device of claim 1, wherein the hardware controller operating with the first virtual machine for accelerating the one or more activities in the first virtual machine, where an activity of the one or more activities corresponds to the anomalous behavior.

11. A network device comprising:

a plurality of virtual machines based on one or more software modules stored within a memory storage device; and

a hardware controller operating with a first virtual machine of the plurality of virtual machines to (i) monitor one or more behaviors of at least the first virtual machine, (ii) accelerate one or more activities in the first virtual machine to reduce an amount of time to detect a time-delayed malware that performs one or more operations to avoid detection by (a) intercepting one or more time-sensitive system calls and (b) modifying the intercepted one or more time-sensitive system calls, (iii) identify at least one behavior of the one or more monitored behaviors as an anomalous behavior, and (iv) detect, based at least in part on the identified at least one anomalous behavior, the time-delayed malware in the first virtual machine.

12. The network device of claim 11, wherein the intercepted one or more time-sensitive system calls comprises a sleep system call having a time parameter, the time parameter is modified so as to accelerate the one or more activities in the first virtual machine.

13. The network device of claim 11, wherein the one or more time-sensitive system calls includes a call to be serviced by a response from an operating system.

14. The network device of claim 13, wherein the hardware controller operating with the first virtual machine for accelerating the one or more activities in the first virtual machine, where the one or more activities correspond to the one or more behaviors.

15. The network device of claim 13, wherein the hardware controller operating with the first virtual machine to accelerate one or more activities in the first virtual machine prior to the hardware controller detecting the time-delayed malware.

16. The network device of claim 11, wherein the hardware controller operating with the first virtual machine to accelerate one or more activities in the first virtual machine, where an activity of the one or more activities corresponds to the anomalous behavior.

17. The network device of claim 11, wherein the hardware controller operating with the first virtual machine to accelerate one or more activities in the first virtual machine, where the one or more activities correspond to the one or more behaviors.

18. A network device comprising:

a plurality of virtual machines based on one or more software modules stored within a memory storage device; and

a hardware controller operating with a first virtual machine of the plurality of virtual machines, the hardware controller accelerating one or more activities in the first virtual machine by at least (i) identifying a time consuming program loop executing in the first virtual machine and monitoring one or more behaviors of the time consuming program loop, (ii) accelerating execution of the time consuming program loop in the first virtual machine to detect a time-delayed malware that is attempting to avoid detection,

wherein the hardware controller further (iii) identifying at least one anomalous behavior of the one or more monitored behaviors, and (iv) detecting, based at least in part on the identifying of the at least one anomalous behavior, the time-delayed malware operating with the time consuming program loop in the first virtual machine.

19. The network device of claim 18, wherein the hardware controller accelerates execution of the time consuming program loop by increasing a priority of execution of the time consuming program loop in the first virtual machine.

20. A method for detecting a presence of malware within data under analysis, the method comprising:

monitoring, by a controller, one or more behaviors that have occurred in response to processing the data under analysis within a first virtual machine;

accelerating, by the controller operating with the first virtual machine, one or more activities in the first virtual machine to detect a time-delayed malware that performs one or more operations to avoid detection, the accelerating of the one or more activities comprises intercepting one or more time-sensitive system calls and modifying one or more responses to the one or more of the time-sensitive system calls;

identifying, by the controller, at least one anomalous behavior in the one or more monitored behaviors; and

detecting, by the controller, the time-delayed malware in the data based, at least in part, on the identified at least one anomalous behavior.

21. The method of claim 20, wherein the one or more time-sensitive system calls includes a call to be serviced by a response from an operating system.

22. A network device comprising:

a plurality of virtual machines operating within the network device; and

a hardware controller in communication with and operating in conjunction with a first virtual machine of the plurality of virtual machines, to (i) monitor one or more behaviors of at least the first virtual machine, (ii) accelerate one or more activities in the first virtual machine, to reduce an amount of time needed to detect a time-delayed malware that performs one or more operations to avoid detection, by at least intercepting one or more time-sensitive system calls and modifying one or more responses to the one or more of the time-sensitive system calls, (iii) identify at least one anomalous behavior in the one or more monitored behaviors, and (iv) detect, based at least in part on the identified at least one anomalous behavior, the time-delayed malware in the first virtual machine.

23. The network device of claim 22, wherein the hardware controller is coupled to a traffic analysis device that analyzes network traffic to determine whether the network traffic includes network communications characteristics of time-delayed malware and provides a copy of the network traffic, including the time-delayed malware, to the hardware controller.

24. The network device of claim 22, wherein the one or more time-sensitive system calls are generated by the time-delayed malware.

25. The network device of claim 24, wherein the one or more time-sensitive system calls comprises a time-of-day system call, and the one or more modified responses to the time-of-day system call by the first virtual machine is to specify a future time.

26. The network device of claim 22, wherein the time-delayed malware comprises one or more computer worms.

27. The network device of claim 22, wherein the controller operating with the first virtual machine accelerates one or more activities in the first virtual machine to reduce a time for detecting the time-delayed malware.

28. The network device of claim 22, wherein the controller accelerates one or more activities in the first virtual machine to reduce a time for detecting the time-delayed malware.

29. The network device of claim 22, wherein the one or more time-sensitive system calls includes a call to be serviced by a response from an operating system.

30. A network device comprising:

a plurality of virtual machines operating within the network device; and

a hardware controller operating with at least a first virtual machine of the plurality of virtual machines to (a) monitor one or more behaviors of at least the first virtual machine of the plurality of virtual machines, (b) accelerate one or more activities in the first virtual machine to detect a time-delayed malware that performs one or more operations to avoid detection by at least (i) intercepting a system call and (ii) modifying a response to the intercepted system call so as to accelerate the one or more activities in the first virtual machine that are occurring during processing of data under analysis by the first virtual machine, (c) identify at least one anomalous behavior in the one or more monitored behaviors, and (d) detect, based at least in part on the identified at least one anomalous behavior, the time-delayed malware in the data under analysis processed by the first virtual machine.

31. The network device of claim 30, wherein the system call includes a sleep system call having a time parameter, the time parameter is modified by the first virtual machine in order to accelerate the one or more activities.

32. The network device of claim 30, wherein the system call is a call to be serviced by a response from an operating system.

33. A network device comprising:

a plurality of virtual machines operating within the network device; and

a hardware controller operating with at least a first virtual machine of the plurality of virtual machines to (a) monitor one or more behaviors of at least the first virtual machine of the plurality of virtual machines, (b) accelerate one or more activities in the first virtual machine to detect a time-delayed malware that performs one or more operations to avoid detection by at least (i) identifying a time consuming program loop executing in the first virtual machine, and (ii) accelerating execution of the time consuming program loop in the first virtual machine by increasing a priority of execution of the time consuming program loop in the first virtual machine, (c) identify at least one anomalous behavior in the one or more monitored behaviors, and (d) detect, based at least in part on the identified at least one anomalous behavior the time-delayed malware in the first virtual machine.

34. A system comprising:

a traffic analysis device configured to receive data over a communication network; and

a network device in communication with the traffic analysis device, the network device to receive suspicious data from the traffic analysis device,

wherein the network device comprises (1) a plurality of virtual machines including a first virtual machine and (2) a controller operating with at least the first virtual machine of the plurality of virtual machines to

(a) monitor one or more behaviors of at least the first virtual machine processing the suspicious data,

(b) accelerate the one or more activities in the first virtual machine to detect a time-delayed malware that performs one or more operations to avoid detection by controlling interception of one or more time-sensitive system calls and modifying one or more responses to the one or more of the time-sensitive system calls,

(c) identify at least one behavior of the one or more behaviors as an anomalous behavior, and

(d) detect, based at least in part on the identified at least one anomalous behavior, the time-delayed malware within the suspicious data processed by the first virtual machine.

35. The system of claim 34, wherein the controller accelerates the one or more activities by controlling interception of one or more time-sensitive system calls initiated by the time-delayed malware and modification of one or more responses to the one or more of the time-sensitive system calls.

36. The system of claim 34, wherein the controller operating with the first virtual machine controlling the first virtual machine to intercept the one or more time-sensitive system calls initiated by the time-delayed malware and modify the one or more responses.

37. The system of claim 34, wherein the controller operating with at least the first virtual machine accelerates the one or more activities in the first virtual machine by intercepting the one or more time-sensitive system calls that comprises a time-of-day system call, and modifying a time value returned by the one or more responses to the time-of-day system call to specify a future time.

38. The system of claim 34, wherein the controller operating with at least the first virtual machine accelerates the one or more activities in the first virtual machine by controlling interception of the one or more time-sensitive system calls that comprises a sleep system call including a time parameter and modifying the time parameter so as to accelerate the one or more activities in the first virtual machine.

39. The system of claim 34, wherein the time-delayed malicious data comprises one or more computer worms.

40. The system of claim 34, wherein the suspicious data contains at least one characteristic associated with time-delayed malware.

41. The system of claim 34, wherein the traffic analysis device is a hardware device that is physically separate from the network device and wherein the traffic analysis device is physically coupled to the communication network.

42. The system of claim 34, wherein the one or more time-sensitive system calls includes a call to be serviced by a response from an operating system.

43. The system of claim 34, wherein the controller operating with at least the first virtual machine to accelerate the one or more activities in the first virtual machine prior to the controller detecting the time-delayed malware.

44. The system of claim 34, wherein the controller operating with at least the first virtual machine to accelerate the one or more activities in the first virtual machine, where an activity of the one or more activities corresponds to the anomalous behavior.

45. The system of claim 34, wherein the controller operating with the first virtual machine to accelerate the one or more activities in the first virtual machine, where the one or more activities correspond to the one or more behaviors.

46. A system comprising:

a traffic analysis device configured to receive data over a communication network; and

a network device in communication with the traffic analysis device, the network device to receive suspicious data from the traffic analysis device,

wherein the network device comprises (1) a plurality of virtual machines including a first virtual machine and (2) a controller operating with at least the first virtual machine of the plurality of virtual machines to

(a) monitor behaviors of at least the first virtual machine,

(b) accelerate one or more activities in the first virtual machine, to reduce an amount of time needed to detect a time-delayed malware that performs one or more operations to avoid detection, by (i) identifying a time consuming program loop executing in the first virtual machine and (ii) accelerating execution of the time consuming program loop in the first virtual machine,

(c) identify at least one anomalous behavior of the monitored behaviors, and

(d) detect, based at least in part on the identified at least one anomalous behavior, the time consuming program loop being associated with the time-delayed malware within the first virtual machine.

47. The system of claim 46, wherein the controller of the network device operating with at least the first virtual machine to accelerate the one or more activities in the first virtual machine prior to the controller detecting the time-delayed malware.

48. The system of claim 46, wherein the controller of the network device operating with at least the first virtual machine to accelerate the one or more activities in the first virtual machine, where an activity of the one or more activities corresponds to the anomalous behavior.

49. The network device of claim 46, wherein the controller of the network device operating with at least the first virtual machine to accelerate the one or more activities in the first virtual machine, where the one or more activities correspond to the behaviors.

50. A network device comprising:

one or more virtual machines, including a first virtual machine; and

a hardware controller operating with the one or more virtual machines to (i) monitor one or more behaviors of at least the first virtual machine during processing of data under analysis, (ii) accelerate one or more activities in at least the first virtual machine to detect a time-delayed malware that performs one or more operations to avoid detection by intercepting one or more time-sensitive system calls and modifying one or more responses to the one or more of the time-sensitive system calls, (iii) identify at least one behavior of the one or more behaviors as an anomalous behavior, and (iv) detect, based at least in part on the identified at least one anomalous behavior, the time-delayed malware within the data.

51. The network device of claim 50, wherein the one or more time-sensitive system calls comprises a time-of-day system call, and the response to the time-of-day system call by the first virtual machine is to specify a future time.

52. The network device of claim 50, wherein the one or more time-sensitive system calls includes a sleep system call having a time parameter, the time parameter is modified so as to accelerate the one or more activities in the first virtual machine.

53. The network device of claim 50, wherein the time-delayed malware includes one or more computer worms.

54. The network device of claim 50, wherein the one or more time-sensitive system calls includes a call to be serviced by a response from an operating system.

55. The network device of claim 50, wherein the hardware controller operating with the first virtual machine accelerates the one or more activities prior to the hardware controller detecting the time-delayed malware.

56. The network device of claim 50, wherein the hardware controller operating with the first virtual machine accelerates the one or more activities, where an activity of the one or more activities corresponds to the at least one anomalous behavior.

57. The network device of claim 50, wherein the hardware controller operating with the first virtual machine accelerates the one or more activities, where the one or more activities correspond to the one or more monitored behaviors.