System and method for detecting anomalous behaviors using a virtual machine environment

• US 9,661,018 B1
• Filed: 05/27/2016
• Issued: 05/23/2017
• Est. Priority Date: 04/01/2004
• Status: Active Grant

• Current Assignee: Fireeye Security Holdings US LLC
• Original Assignee: FireEye, Inc.
• Inventors: Aziz, Ashar
• Primary Examiner(s): Dolly, Kendall
• Application Number:
  US15/167,645
• Time in Patent Office: 361 Days
• CPC Class Codes: G06F 2009/45587 Isolation or security of virtual machine instances
• CPC Class Codes: G06F 2009/45591 Monitoring or debugging support
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: H04L 63/0245 Filtering by information in the payload
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms
• CPC Class Codes: H04L 63/1458 Denial of Service
• CPC Class Codes: H04L 63/1491 using deception as countermeasure, e.g. honeypots, honeynets, decoys or entrapment

Abstract

A network device for detecting malware is described. The network device features a memory storage device and a controller. The controller operating in cooperation with one or more virtual machines that are based on software modules stored within the memory storage device. The controller is configured to (i) monitor behaviors of at least a first virtual machine of the one or more virtual machines processing data received over a network, (ii) identify at least one anomalous behavior that includes either a communication anomaly or an execution anomaly, and (iii) detect, based on the identified at least one anomalous behavior, a presence of malware in the first virtual machine in response to identifying the at least one anomalous behavior.

Claims

1. A network device comprising:

a memory storage device; and

a hardware controller operating in cooperation with one or more virtual machines that are based on software modules stored within the memory storage device, the hardware controller to (i) select an orchestration pattern based on a type of data received over a network for analysis, the orchestration pattern identifies at least one or more ports accessible by at least a first virtual machine of the one or more virtual machines during processing of the data and coordinates network activities by the one or more virtual machines based on the selected orchestration pattern, (ii) monitor behaviors of at least the first virtual machine of the one or more virtual machines processing data received over the network, (iii) identify at least one anomalous behavior that includes either a communication anomaly or an execution anomaly, and (iv) detect, based on the identified at least one anomalous behavior, a presence of malware in the first virtual machine in response to identifying the at least one anomalous behavior that includes one or more accesses of a port other than the one or more ports identified by the orchestration pattern.

2. The network device of claim 1 being communicatively coupled to the network.

3. The network device of claim 1, wherein the hardware controller is further configured to generate a signature to detect the malware in network traffic propagating over the network that is different from the data received over the network.

4. The network device of claim 1, wherein the malware is a computer worm.

5. The network device of claim 1, wherein the one or more virtual machines operate as a virtual computer network and the one or more virtual machines includes a driver software that supports message exchanges between the one or more virtual machines.

6. The network device of claim 5, wherein the hardware controller monitors the behaviors of at least the first virtual machine by logging data from messages initiated by the first virtual machine.

7. The network device of claim 5, wherein the hardware controller being further configured to automatically update a software profile associated with any of the one or more virtual machines.

16. The network device of claim 1, wherein the orchestration pattern includes an expected behavior in accessing the one or more ports by the first virtual machine.

8. A system comprising:

a traffic analysis device configured to receive data over a communication network and identify a type of data received over the communication network; and

a network device in communication with the traffic analysis device, the network device comprises

a memory, and

a controller being one or more software modules contained in the memory and, when executed, operates in cooperation with one or more virtual machines that are based on software modules stored within the memory, the controller to (i) select an orchestration pattern, based on the type of data identified by the traffic analysis device, that identifies at least one or more ports accessible by the one or more virtual machines during processing of the data and coordinates network activities by the one or more virtual machines based on the selected orchestration pattern, (ii) monitor behaviors of at least a first virtual machine of the one or more virtual machines processing the data received from the traffic analysis device, (iii) identify at least one anomalous behavior that includes either a communication anomaly or an execution anomaly, and (iv) detect, based on the identified at least one anomalous behavior, a presence of malware in the first virtual machine in response to identifying the at least one anomalous behavior that includes one or more accesses of a port other than the one or more ports identified by the orchestration pattern.

9. The system of claim 8, wherein the controller of the network device is further configured to generate a signature to detect the malware in network traffic propagating over the communication network that is different from the data received by the traffic analysis device.

10. The system of claim 8, wherein the malware is a computer worm.

11. The system of claim 8, wherein the one or more virtual machines within the network device operate as a virtual computer network and the one or more virtual machines includes a driver software that supports message exchanges between the one or more virtual machines.

12. The system of claim 11, wherein the controller of the network device monitors the behaviors of at least the first virtual machine by logging data from messages initiated by the first virtual machine.

13. The system of claim 8, wherein the controller of the network device being further configured to automatically update a software profile associated with any of the one or more virtual machines.

14. The system of claim 8, wherein the traffic analysis device is configured to identify the type of the data received from the communication network before propagating the data to the network device.

15. The system of claim 14, wherein the controller of the network device selects the orchestration pattern that includes an expected behavior of the first virtual machine of the one or more virtual machines.

17. A system comprising:

a hardware traffic analysis device configured to receive data over a communication network and to selectively filter the data and output a first portion of the data received over the communication network; and

a network device in communication with the hardware traffic analysis device, the network device comprises

a memory storage device, and

a hardware controller operating in cooperation with one or more virtual machines that are based on software modules stored within the memory storage device, the hardware controller to (i) select an orchestration pattern that is based on a type of the data received over the communication network and identifies at least one or more ports accessible by at least a first virtual machine of the one or more virtual machines during processing of the data and coordinates network activities by the one or more virtual machines based on the selected orchestration pattern, (ii) monitor behaviors of at least the first virtual machine of the one or more virtual machines processing the first portion of the data received as output from the traffic analysis device, (iii) identify at least one anomalous behavior that is part of the monitored behaviors as either a communication anomaly or an execution anomaly, and (iv) detect, based on the identified at least one anomalous behavior, a presence of malware in the first virtual machine in response to identifying the at least one anomalous behavior that includes one or more accesses of a port other than the one or more ports identified by the orchestration pattern.

18. The system of claim 17, wherein the traffic analysis device selectively filters the data by precluding a second portion of the data from propagating from the communication network to the hardware controller.

19. The system of claim 17, wherein the traffic analysis device selectively filters the data by precluding Internet Protocol (IP) data packets from being routed from the communication network to the hardware controller.

20. A network device comprising:

a memory including one or more virtual machines; and

a controller operating in cooperation with one or more virtual machines that are based on software modules stored within the memory, the controller to (i) select an orchestration pattern based on a type of data received over a network for analysis, the orchestration pattern identifies at least one or more ports accessible by at least a first virtual machine of the one or more virtual machines during processing of the data and coordinates network activities by the one or more virtual machines based on the selected orchestration pattern, (ii) monitor behaviors of at least the first virtual machine of the one or more virtual machines processing data received over the network, (iii) identify at least one anomalous behavior that includes either a communication anomaly or an execution anomaly, and (iv) detect, based on the identified at least one anomalous behavior, a presence of malware in the first virtual machine in response to identifying the at least one anomalous behavior that includes one or more accesses of a port other than the one or more ports identified by the orchestration pattern.

21. The network device of claim 20 being communicatively coupled to the network and the controller being a hardware controller.

22. The network device of claim 20, wherein the controller is further configured to generate a signature to detect the malware in network traffic propagating over the network that is different from the data received over the network.

23. The network device of claim 20, wherein the malware is a computer worm.

24. The network device of claim 20, wherein the one or more virtual machines operate as a virtual computer network and the one or more virtual machines includes a driver software that supports message exchanges between the one or more virtual machines.

25. The network device of claim 24, wherein the controller monitors the behaviors of at least the first virtual machine by logging data from messages initiated by the first virtual machine.

26. The network device of claim 24, wherein the controller being further configured to automatically update a software profile associated with any of the one or more virtual machines.