Systems and methods for automatically detecting backdoors

• US 8,990,944 B1
• Filed: 02/23/2013
• Issued: 03/24/2015
• Est. Priority Date: 02/23/2013
• Status: Active Grant

• Current Assignee: Fireeye Security Holdings US LLC
• Original Assignee: FireEye, Inc.
• Inventors: Manni, Jayaraman, Singh, Abhishek
• Primary Examiner(s): Arani, Taghi
• Assistant Examiner(s): CHANG, KENNETH W
• Application Number:
  US13/775,175
• Time in Patent Office: 759 Days
• Field of Search: 726/24
• US Class Current: 726/24
• CPC Class Codes: G06F 21/53 by executing in a restricted environment, e.g. sandbox or secure virtual machine
• CPC Class Codes: G06F 21/56 Computer malware detection or handling, e.g. anti-virus arrangements
• CPC Class Codes: G06F 21/566 Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities
• CPC Class Codes: G06F 21/577 Assessing vulnerabilities and evaluating computer system security

Abstract

Techniques for detecting malicious attacks may monitor activities during execution of content (e.g., network downloads and email) to identify an instruction backdoor on a computer. An instruction backdoor is malware that opens a port for listening for instructions from a remote server. Such instructions may trigger activation of malicious activity. These techniques employ virtual machines that, in a controlled and monitored environment, permit content, when loaded and executed, to open ports. When those ports are not the authorized ones used for communications by known programs installed on the computer, an instruction backdoor is discovered.

Claims

1. A computer-implemented method for detecting a backdoor to a computer system, comprising:

inspecting content comprising a malware candidate;

configuring a virtual machine provided by a controller, the virtual machine executing an operating system instance and a program instance;

executing the content comprising the malware candidate in the virtual machine;

detecting whether the content comprising the malware candidate is listening to a port in the virtual machine, and identifying a port number associated with the port at which the content comprising the malware candidate is listening;

determining whether the port number of the port at which the content comprising the malware candidate is listening is a black port by accessing a first repository that comprises a first data structure of port numbers of ports at which known malware has been detected as listening to determine whether the port number of the port is in the first data structure of the first repository, and initiating an access to a second repository that comprises a second data structure of port numbers of ports that are legally accessed by one or more programs including the program instance to determine whether the port number of the port is in the second data structure of the second repository;

in response to the port number of the port being present in the first data structure of the first repository, reporting the content comprising the malware candidate as including the backdoor.

2. The computer-implemented method of claim 1, wherein detecting comprises intercepting system calls from the program instance executing within the virtual machine to the operating system instance executing within the virtual machine, the system calls requesting the operating system instance bind an identifier of the program instance to a port number of a port on which the content comprising the malware candidate is to listen.

3. The computer-implemented method of claim 1, wherein detecting comprises reading data from a memory included in the controller specifying a program identifier of a program instance executing within the virtual machine and a port number corresponding to a port at which the content comprising the malware candidate is to listen.

4. The computer-implemented method of claim 1, wherein the content comprising the malware candidate is associated with a score related to a probability that the network content includes malicious network content;

and further comprising setting the score upon determining that the port at which the content comprising the malware candidate is listening is a black port to a level sufficient to indicate a high probability of malware.

5. The computer-implemented method of claim 1, wherein the first data structure of the first repository includes a blacklist of ports at which known malware has been detected as listening.

6. The computer-implemented method of claim 5, wherein the second data structure of the second repository includes a whitelist.

7. The computer-implemented method of claim 1, wherein the backdoor is an instruction backdoor that includes the malware candidate that opens the port, that is a communication port of the computer system, for listening for instructions or other communications from an external malicious computer system or program.

8. The computer-implemented method of claim 1, wherein the first repository does not contain port numbers of ports that are legally accessed by the program instance.

9. The computer-implemented method of claim 6, further comprising updating the blacklist with the port number of the port at which the content comprising the malware candidate is listening in the event that the comparing of the port number of the port to the port numbers in the blacklist and the port numbers in the whitelist indicates that the port number of the port is not in the first repository or the second repository.

10. The computer-implemented method of claim 6, further comprising indicating that further testing is required in the event that the port number of the port at which the content comprising the malware candidate is listening is in both the first repository and the second repository.

11. The computer-implemented method of claim 6, wherein reporting comprises reporting that the content comprising the malware candidate includes the backdoor and thereby verifying that the malware candidate should be treated as comprising malware, but only in response to determining that the port number of the port at which the content comprising the malware candidate is listening is in the first repository without being in the second repository.

12. The computer-implemented method of claim 1, further comprising generating a malware signature for the content comprising the malware candidate that includes the backdoor and sharing the malware signature with at least one malware detection system.

13. The computer-implemented method of claim 12, further comprising uploading the malware signature to a malware management system.

14. A system for detecting a backdoor to a computer system, comprising:

a controller comprising a processor and a memory coupled with the processor and configured to store a repository for content comprising a malware candidate, the controller further comprising

a) a configurator adapted to inspect the content comprising the malware candidate and configure a virtual machine in response to the inspection, the virtual machine configured to execute an operating system instance and a program instance,

b) a virtual machine configured to execute the content comprising the malware candidate, and

c) a backdoor analyzer configured to (i) detect whether the content comprising the malware candidate is listening at a port in the virtual machine, (ii) identify a port number associated with the port at which the content comprising the malware candidate is listening, (iii) determine whether the port number of the port at which the content comprising the malware candidate is listening is a legal port by initiating an access a first repository comprising at least one data structure containing a plurality of port numbers of ports that are legally accessed by the program instance so as to determine whether the port number of the port corresponds to one of the plurality of port numbers in the first repository, and (iv) determine whether the port number of the port at which the content comprising the malware candidate is listening is an illegal port by accessing a second repository comprising at least one data structure containing a plurality of port numbers of ports at which known malware has been detected as listening and comparing the port number of the port to the port numbers in the second repository; and

an event generator coupled with the controller and an output interface, and configured to generate an event indicating, if an illegal port, that the content comprising the malware candidate includes the backdoor and identifying that the malware candidate should be treated as comprising malware.

15. The system of claim 14, wherein the backdoor analyzer further comprises a port monitor communicatively coupled with the virtual machine for intercepting one or more system calls from the program instance executing within the virtual machine to the operating system instance executing within the virtual machine, the system calls requesting for the program instance bind to and listen to a port provided by the operating system instance.

16. The system of claim 14, wherein the backdoor analyzer further comprises a port monitor for reading data from a memory included in the controller specifying a program instance identifier corresponding to the program and a port number corresponding to a port at which the content comprising the malware candidate is listening.

17. The system of claim 14, wherein the content comprising the malware candidate is associated with a score related to a probability that the network content includes malicious network content;

and wherein the controller is further configured to increase the score upon determining that the port at which the content comprising the malware candidate is listening is not a legal port, the increase being sufficient to indicate a high probability of malware.

18. The system of claim 14, wherein the second repository comprises the at least one data structure operating as a blacklist of ports at which known malware has been detected as listening.

19. The system of claim 14, wherein the second repository comprises the at least one data structure operating as a blacklist of ports at which known malware has been detected as listening;

the blacklist being associated with a software profile corresponding to that of the configured virtual machine.

20. The system of claim 14, wherein the backdoor is an instruction backdoor that comprises malware that opens the port for listening for instructions or other communications from an external malicious computer system or program.

21. The system of claim 14, wherein the second repository includes a plurality of entries each corresponding to a port number and the entries do not include port numbers of ports that are legally accessed by programs identified by a software profile associated with the configured virtual machine.

22. The system of claim 14, wherein content within the at least one data structure of the second repository comprises a blacklist and content within the at least one data structure of the first repository comprises a whitelist.

23. The system of claim 22, wherein backdoor analyzer is configured to update the blacklist with the port number of the port in the event that the port number of the port is in neither the second repository nor the first repository.

24. The system of claim 22, wherein the backdoor analyzer is coupled with a user interface and is configured to report via the user interface that further testing is required in the event the port number of the port is in both the blacklist of the second repository and the whitelist of the first repository.

25. The system of claim 22, wherein the backdoor analyzer is configured to report via the user interface in the event the port number of the port is in the blacklist of the second repository without being in the whitelist of the first repository that the content comprising the malware candidate includes the backdoor and thereby verifying that the malware candidate should be treated as comprising malware.

26. The system of claim 14, wherein the backdoor analyzer further comprising a signature generator for generating a malware signature for the content comprising the malware candidate that includes the backdoor and sharing the malware signature with at least one malware detection system.

27. The system of claim 14 further comprising a network interface, and wherein the controller is configured to upload, via the network interface, a malware signature generated by the backdoor analyzer to a malware management system.

28. A system for discovering an instruction backdoor to a digital device to detect malware, the system comprising:

a controller comprising a memory and at least one virtual machine configured with a software profile which identifies at least one program and an operating system to mimic aspects of the digital device and, in a controlled and monitored environment, load and execute one or more objects; and

a backdoor analyzer coupled for communication with the controller, a first repository and a second repository, the backdoor analyzer is configured to detect, during execution by the virtual machine of the one or more objects, a port opened by the one or more objects for listening for communications from an external system that is external to the digital device,

wherein the backdoor analyzer is further configured to identify a port number for the port opened for listening, check whether the port number is associated with a black port by at least checking whether the port number for the opened port corresponds with an entry on a blacklist stored in the first repository of known malware, checking whether the port number for the opened port is absent from an entry on a whitelist stored in the second repository of ports used by the at least one program of the software profile, and responsive to the port number corresponding to the entry on the blacklist, reporting that the one or more objects include malware operating as the instruction backdoor.

29. The system of claim 28, wherein the backdoor analyzer is further configured to update the blacklist with the port number for the opened port in response to the port number being absent from the first repository and the second repository.

30. The system of claim 28, wherein the backdoor analyzer is configured to indicate that further testing is required in the event the port number of the opened port is in both an entry of the blacklist of the first repository and an entry of the whitelist of the second repository.

31. The system of claim 28, wherein a determination by the backdoor analyzer that the one or more objects opens a black port is sufficient predicate by itself to indicate that the one or more content objects comprise malware.

32. The system of claim 28, wherein the instruction backdoor comprises the malware that opens the opened port for listening for instructions or other communications from the external system.

33. The system of claim 28, wherein the virtual machine further comprises a port monitor communicatively coupled with the backdoor analyzer for intercepting system calls from a process executing within the virtual machine to an operating system instance executing within the virtual machine, the system calls requesting for the process to bind to and listen to a port provided by the operating system instance.

34. The system of claim 28, wherein the virtual machine further comprises a port monitor coupled with the backdoor analyzer for reading data from a memory included in the controller specifying a process number corresponding to the process and a port number corresponding to a port at which the one or more content objects are listening.

35. A non-transitory computer readable storage medium having stored thereon instructions executable by a processor for performing a method, the method comprising the steps of:

executing a program instance to process the content;

detecting whether the program is listening to a port, and identifying a port number associated with the port;

determining whether the port number of the port at which the program instance is listening is an authorized port, the determining comprises (i) accessing a blacklist repository that includes a first data structure of port numbers of ports at which known malware has been detected as listening, (ii) initiating an access of a whitelist repository that includes a second data structure of port numbers of ports that are legally accessed by programs, and (iii) determining whether the port number of the port is in the first data structure of the blacklist repository or is in the second data structure of the whitelist repository; and

in response to (a) the port number of the port being within the first data structure of the blacklist repository or (b) the port number of the port being absent from the first data structure of the blacklist repository and the second data structure of the whitelist repository, reporting the content as containing a backdoor.

36. The non-transitory computer readable storage medium of claim 35, wherein the processor detecting whether the program is listening to the port comprises intercepting system calls from the program instance to an operating system instance, the system calls requesting the operating system instance bind an identifier of the program instance to a port number of the port on which the program instance is listening.

37. The non-transitory computer readable storage medium of claim 35, wherein the processor detecting whether the program is listening to the port comprises reading data from a memory specifying a program identifier of a program instance and a port number corresponding to a port at which the program instance is listening.

38. The non-transitory computer readable storage medium of claim 35, wherein the processor further updating the first data structure of the blacklist repository with the port number of the port at which the content comprising the malware candidate is listening.

39. The non-transitory computer readable storage medium of claim 35, wherein the backdoor is an instruction backdoor that includes a malware that opens the port, which is a communication port of a digital device including the non-transitory computer readable storage medium, for listening for instructions or other communications from an external malicious computer system or program.